It broke while you weren’t watching

The delayed and failed calls in our development exercise were arranged on purpose. A failure in use arrives with less explanation. Someone says the site is broken, and we need to find the request they mean and what happened to it. That is much easier if the program has left a useful record.

The silent catch

The shortest way to throw the account away is a catch block with nothing inside it.

try {
  await saveDraft(userId, draft);
} catch (e) {
  // ignore
}

The surrounding operation can continue as if saving succeeded. Without a test for the failure path, that loss may be invisible to the suite as well. Caught sounds close enough to handled that people stop there. But catching an exception and doing nothing with it is your program deciding, on your behalf, that you need not know what went wrong. People write empty catches because they make the red text go away, and the red text was the feature.

Handling an error means doing one of three things: recover in a way you can describe, tell the caller with an explicit error, or let the error travel up to a place that will report it. For failures you will need to diagnose, record the event once at the layer with useful context, without logging the private draft itself.

Before reading on, search your own codebase for catch blocks that are empty, or that contain only a comment or only a print. How many are there?

Logs you can ask questions

A log is a record of what somebody thought, in advance, to write down. Print lines are enough for a day. Once there are tens of thousands of lines, you no longer want to read the file. You want to ask it a question, and that is why the important values belong in the line as fields rather than as prose:

{"level":"error","msg":"save draft failed","request_id":"7f3a9c","route":"POST /drafts","user_id":"u_1042","error":"connection reset"}

Prose lines you scroll through. Fields you can filter.

The request_id is a correlation ID. Generate one when a request enters your system and attach it to every log line written while handling that request. Pass it along in a header to any other service you call, and show it on your error pages. Then a visitor's screenshot, your API log and your worker log can be joined into one story.

Just as important is what never goes into a log: passwords, session tokens, API keys, full card numbers, identity documents, and whole request bodies. Logs get copied, shipped to other services and kept longer than databases, and more people read them. When an internal id is enough, log the id, not the email.

Someone has to be told

Logs nobody reads are a diary. Errors need routing. Send unhandled exceptions to an error tracker or a channel someone actually watches, grouped so that one failure appears once with a count instead of a thousand separate times.

Alerts are for user impact, not for every exception. Checkout failing for a noticeable share of attempts, a queue that has stopped draining, a scheduled job that has not succeeded since yesterday: those should wake someone. A single timeout that succeeded on retry should not. An alert that fires constantly teaches people to ignore it.

Getting back to working

When something breaks and you cannot diagnose it yet, the first move is often to restore service and diagnose afterwards. Rolling back a deploy helps when the deploy caused the problem. It does not help when a migration has already changed data, when an external provider is down, or when the bug has been there for months. Those incidents need other tools. A feature flag turns one feature off while the rest keeps serving. A backup restore works if you have actually practised it. Sometimes the answer is a forward fix. Know which incidents each tool covers before you need it.

A record the agent does not keep

I built rabadon because long-running coding agents can drift, break something on the way, and still report done. The agent passes its own tests, so tests alone did not catch the drift. What was missing was a record kept by someone other than the agent.

rabadon's audit log is an HMAC chain. The previous-record field links each entry to the hash of the preceding entry. In HMAC mode, that hash is keyed; a head sidecar records the final hash and count. Editing, removing or reordering a record breaks the chain, and cutting records off the end no longer matches the head. It is worth being precise about the limit. The key lives on the same machine, under the same user, so anyone with that user's access can recompute the whole chain. The chain makes quiet edits detectable, not impossible. The record provides something to inspect independently of the agent's summary. Protecting the key and verifier is a separate boundary.