When one of your own servers breaks, you usually find out within minutes. Its metrics turn red, its logs fill with errors, and someone gets paged. When software you shipped breaks somewhere you don’t run it (a customer’s server, an install behind someone else’s firewall, an employee’s laptop), it usually goes quiet. And quiet looks the same as a machine that’s simply switched off.

That’s why I think monitoring for these systems has to be built into the software itself, not added on top later. A system should be able to explain what happened to it, even if nobody was listening at the time.

Limits of central monitoring

Cloud monitoring assumes your servers are always reachable. A central system collects metrics and logs all the time, so when something fails the evidence is already in one place. Software running somewhere you don’t control can’t promise that. It goes offline, sits behind firewalls and proxies, gets restarted at odd times, and loses connectivity halfway through an update. And the failures that matter most, like an update that leaves the software unable to start, are the ones that stop it from reporting anything.

Crash reporters cover the program that dies while running. But they usually ship and update together with the program, so an update that leaves a half-installed version, or one that never starts, can take the reporting down with it. Asking for logs after the fact rarely helps either: they’re often missing, incomplete or too late to be useful.

The closest thing to a solution lives in the operating system. Phones keep two copies of the system, count failed boots, and fall back to the old copy when the new one won’t start, so a bad update can’t brick the device. But the record there says only that a boot failed, not which step of your update did.

Two rules

The record has to outlive the code that failed. Databases solved this decades ago with write-ahead logging: write down what you’re about to do, then do it. If the process dies halfway, whoever opens the database next reads the log and knows what happened. Shipped software can do the same. Before an update, note “moving from version 11 to 12” somewhere the update won’t touch. If version 12 never comes up, the updater or the previous version reads the note and reports “version 12 never came up” instead of nothing.

Erlang’s “let it crash” makes the same point: the part that failed is the worst place to handle the failure, so let something else do it. A short history of versions and errors, sent once the system is back online, is enough. It doesn’t need full logs.

Watch the change, not the silence. If every install checks in on a schedule and says which version it’s on, one missed check-in still tells you nothing. A machine that’s broken, switched off or just unreachable looks the same from the outside. What tells you something is a comparison: are installs on the new version missing check-ins more often than those still on the old one? That’s how the Google SRE workbook judges a canary release, against a control group rather than against silence.

What the record holds

Keep it small: the version the system was on, the version it was moving to, the last step it reached, an error code if there was one, and timestamps. That answers most “what happened?” questions, and a few lines written before the step are far easier to get back afterwards than a full log. Leave out user data and anything you don’t need to investigate. Whoever runs the machine should be able to read the report and find nothing surprising in it.

The version 12 rollout

Say version 12 rolls out to every install over a few days.

Without a record, the first sign is nothing at all. Some installs stop checking in, which looks like normal churn. A week later support tickets start arriving, each from a different setup, and nobody can tell whether they’re related. Engineers ask for logs and get partial ones, from machines that have already been restarted or rolled back by hand.

With a record, it plays out differently. On the first day, installs on version 12 miss check-ins noticeably more often than those still on version 11. The ones that come back report “moving from 11 to 12, reached step 3 (database migration), error 23”. Most of them failed at the same step, so the cause is narrowed down before the first ticket arrives. You pause the rollout, fix the migration, and the installs that fell back to version 11 try again.

None of this needed access to anyone’s machine.

The trade-off

The report format becomes a contract: every version has to write it in a way the updater and your backend can still read, so changing it takes the same care as changing a public API. Whatever reads the record after a failure, usually the updater, becomes the one part that must keep working, so it has to stay small and change rarely. And every install now sends something home on a schedule, which whoever runs the machine will reasonably ask about.

In return, support can tell a broken install from one that’s switched off, and engineers can tell which version and which step caused a failure.

Takeaways

• For software you don’t run yourself, the failures that matter most are the ones that stop it from reporting. Central monitoring and crash reporting alone will miss them.
• Write the plan down before any risky step, where the update can’t touch it, and let something that didn’t change report on it.
• One missed check-in means nothing. Missed check-ins on the new version, compared with the old one, mean something.
• Keep the record small, and treat its format like a public API.

Your own servers tell you when they’re broken. Software you ship has to be built to tell you later.