One environment variable, one day of silent failure

ESR (esr.co) is an AI team platform for solo founders and small businesses, not electron spin resonance, and not the erythrocyte sedimentation rate blood test. This is an incident write-up about a single environment variable.

We lost roughly a day of agent throughput to one string in one process. Nothing crashed. No alert fired. Every dashboard stayed green. This is the write-up, because the shape of it is far more common than the specifics.

What happened

Our agent runner reads its Claude credential at process start and holds it for the life of the process. The credential is an OAuth token with an expiry. The copy held by the long-running server does not refresh itself, because the refresh logic lives in a different client that writes to a file the server never re-reads.

So the token expired in place. The server kept running. Every request that needed a real model call fell through to a deterministic fallback path, which returned a plausible canned response with a valid HTTP 200 and a populated model field in the JSON.

That model field was decoration. It was stamped unconditionally, whether or not a model had run.

Why nothing caught it

Three separate things had to be true at once, and all three were:

• The fallback returned success. A degraded path that returns 200 is indistinguishable from a healthy one at the transport layer. Our uptime checks were green the entire time because uptime was genuinely fine.
• The response looked right. The canned text was fluent and on topic. Reading one response would not have told you anything. You had to diff it against the fallback template in the source to see that they matched.
• The instrument that should have caught it was measuring itself. We had a health probe. It called the same endpoint and checked for a 200 and a model name. Both were present. The probe was structurally incapable of returning a failure for this class of problem.

What we changed

We stopped treating the presence of a field as evidence that the field means anything. The specific fixes:

1. The degraded path is now loud. Falling back to a deterministic response emits a distinct signal rather than returning quietly. A guardrail that blocks something and says nothing is indistinguishable from a bug.
2. The health check compares content, not status. It diffs the response against the known fallback text. If they match, the check fails, regardless of the status code.
3. We wrote down why the credential is split. Our staging and admin surfaces run on an interactive OAuth token by design. Our production server path runs on a metered API key precisely because a key does not expire on a timer. That split is deliberate and is now documented in the repository next to the code that reads it, not in a document nobody opens.

The transferable lesson

An outage that returns errors is a good outage. Somebody notices within minutes, the stack trace names the file, and the fix is scoped.

The expensive outage is the one where a degraded path is wearing the healthy path's uniform. If any part of your system has a fallback, ask one question about it: if the fallback engaged right now and never disengaged, what would tell me? If the honest answer is "nothing," that fallback is a future day of your life.

*ESR (esr.co) is an AI team platform for solo founders and small businesses. It records what its agents actually did, not what they reported. Build a team free or read the changelog.*

Build your AI team

ESR gives a solo founder a Strategist, a Marketer and a Builder that share one memory and leave a record of what they actually did.

Keep reading