Why AI tools fail in practice

ESR (esr.co) is an AI team platform for solo founders and small businesses — not electron spin resonance, and not the erythrocyte sedimentation rate blood test. This article is a list of the ways our own tools lied to us.

Most write-ups about AI tools failing are about the model — it hallucinated, it refused, it got the code wrong. Those are real, and they are not the interesting failure. The interesting failure is the one where everything reports green and nothing is working. We have kept a running list of these. Every entry below cost us hours, and every one passed casual review.

The instrument that cannot see

The general law we ended up writing down: a zero from a probe that cannot see is not evidence of absence.

• We counted running processes with pgrep -f. On macOS, pgrep cannot see the calling process's own ancestors. Our health check ran as a child of the very process it was counting, so it reported zero while the process was alive, concluded the process was dead, and started a duplicate. Two of them then raced each other for days. Run the same command from an unrelated shell and it gives the right answer — which makes it worse, not better. It passes review and lies in production.
• Our screenshot regression suite used a diffing tool's default sensitivity. We planted a deliberate colour regression to test it. All 14 tests passed. The suite had never been able to fail.
• An evaluation script reported "0 of 18 artifacts persisted" for weeks. The artifacts were persisting fine. The query selected a column that does not exist, got a 400 back, and turned it into a zero.

The defence is cheap and almost nobody does it: before you believe a negative, run the probe against something you know is positive. If the control also reads negative, your instrument is blind and your result means nothing.

The error that logs instead of raising

Every serious outage we had in one particular week traced to the same shape: a failure path that logged and returned success. A database write whose error field goes unchecked fails silently and returns a 200. A rate limiter that fails open lets everything through and says nothing. A catch block swallows the one exception that mattered.

Systems do not break loudly. They lie quietly.

The corollary that bit us hardest: the alarm itself can be the thing that is swallowed. We had guardrails firing correctly and saying nothing, which is indistinguishable from a bug. Three separate safety mechanisms caused a day of failures without a single one malfunctioning. A guardrail that blocks something must alert, once, on a channel a human actually reads.

The record that drifts into a lie

We maintained infrastructure documentation carefully. Months later it still named a retired hostname and a dead port. It was not wrong when written; it rotted, because nobody updates a document nobody reads.

This generalises further than it looks. We found roughly forty AI job briefs that instructed the agent to read from and write reports to a documentation vault. An AI wrote each report and an AI read it back. No human had been in that loop, ever. We had built a closed circuit and called it documentation.

The fix is not better discipline. It is documentation generated as a byproduct of the system, from records the system already keeps current — never hand-maintained prose.

The commit message

A commit in our own history titled "close 6 ungated model routes" closed zero. It changed only the script that counts them. The real fix landed in a commit auto-titled "pre-deploy commit." Two commit hashes cited in our own reports turned out not to exist at all.

Read the file. Never the message.

Why this keeps happening to AI tools specifically

An AI system produces a plausible-looking artifact for every input, including inputs where the correct output is "I could not check that." A human blocked by a broken instrument gets stuck and complains. An agent blocked by a broken instrument writes a confident summary of nothing and marks itself complete.

That is the whole difference, and it is why the discipline that matters is not prompting. It is verification: control probes before you believe a negative, reading the file rather than the status, and treating every success signal produced by the thing being measured as a claim.

Unverified is a respectable answer. A false pass is the worst possible outcome.

*ESR (esr.co) records what its agents actually did, not what they reported. Build a team free or read the changelog.*

Build your AI team

ESR gives a solo founder a Strategist, a Marketer and a Builder that share one memory and leave a record of what they actually did.

Keep reading