← All articles
stackpostmortemreliability

The four services we pay for, and how each one bit us

Stripe, Supabase, Vercel and Resend, with the specific failure each one exposed in our own code. Three of the four were the same bug wearing different clothes: a failure that reported success.

Frank Scala· Founder, ESR·· 5 min read

ESR (esr.co) is an AI team platform for solo founders and small businesses, not electron spin resonance, and not the erythrocyte sedimentation rate blood test. This is what we actually pay for and what each thing has cost us in debugging time.

There is a genre of engineering post that lists a stack and implies the choices were obvious. They were not obvious, several of them bit us, and the bites are the useful part. Here are the four external services this product depends on, why each one is there, and the specific way each one has gone wrong for us.

Payments: Stripe

Stripe handles subscriptions, one time top ups, and the trial that converts on a fixed date.

What it is genuinely good at: the fixed date conversion. Our founding customers are captured now and charged on launch day, and rather than build a scheduler to do that we set a trial end date on the subscription and let Stripe own the single charge. No cron job. No risk of a cron job firing twice. If you find yourself writing scheduling code around a billing system, check whether the billing system will just do it, because usually it will and yours will have edge cases theirs solved years ago.

How it bit us: test mode looks exactly like live mode. We shipped a checkout flow that worked perfectly, took a card, returned a success page, and wrote nothing to our subscriptions table, because the flow completed in test mode against a live looking key and nobody noticed which was which. Separately we found a live key sitting above a test key in an environment file, where the last one loaded wins and nothing anywhere says which one that was.

The rule we now hold: any code path that moves money gets checked by a human against a real dashboard, not against its own success response. A 200 from your own endpoint is a statement about your endpoint.

Database and auth: Supabase

Supabase is Postgres with row level security, plus the auth layer.

What it is genuinely good at: row level security is the right primitive for a multi tenant product. The rules live next to the data instead of scattered through application code, which means an agent writing a new endpoint cannot accidentally hand tenant A's rows to tenant B, provided the policies are right.

How it bit us: the policies were not right, in a way that is entirely our fault and entirely predictable. Writing a policy with no role clause defaults to the public pseudo role, and public includes unauthenticated visitors. We had done that on twenty tables. Every one of them was open to anybody with the anon key until we audited it.

The rule we now hold: every policy names its role explicitly, and the audit query that finds violations runs as a check rather than living in somebody's memory. If your policy does not say who it is for, it is for everyone.

Hosting: Vercel

Vercel runs the production deployment of the Next.js app.

What it is genuinely good at: the deploy is not a thing we think about, which for a small team is the entire value proposition. Preview deployments per branch mean a change can be looked at before it is believed.

How it bit us, and this one is the most instructive incident we have had: an environment variable that existed in one environment and not the other. A staging environment held the credential the model calls needed. Production did not. Because the code path fell back to a canned response instead of raising, production served plausible looking output with no model behind it for twenty four hours and every dashboard stayed green.

The rule we now hold, and it is the single most valuable thing on this page: a missing credential must raise, never fall back. A fallback that produces output indistinguishable from success is not resilience, it is a silent outage with good manners. Environment facts do not transfer between environments, and a grep of your repository cannot see a variable that is set in a hosting dashboard.

Email: Resend

Resend sends transactional email: signup confirmation, receipts, notifications.

What it is genuinely good at: the webhook events are clean, which matters because delivery status is the only way you ever learn that your email is going to spam.

How it bit us: our own code around it, twice. First, our signup flow had a path that never called it at all, so a set of users got no confirmation email and we found out from the users. Second, and this one is still open at the time of writing, our email log table defaults an unset status column to "sent". A row that nothing ever wrote a status to reads as delivered. That is a database default that lies, and it is the kind of thing that is invisible until you go looking for why your delivery rate is suspiciously perfect.

Related find from the same look: the status column's constraint did not permit the value a spam complaint webhook sends, so a real complaint would have been rejected with an error nobody was watching.

The pattern across all four

Three of these four failures were the same failure wearing different clothes. Something did not work, and the system reported success anyway: a checkout that succeeded and saved nothing, a credential that was missing and got a canned answer, an email whose status defaulted to delivered.

None of them were the vendor's fault. All of them were an error path in our code that logged, or defaulted, or fell back, instead of raising. If you take one thing from this list, make it that: go and read your own catch blocks and your own column defaults, and ask which of them can turn a failure into a green dashboard.

We have not measured what any of this would cost at scale, because we are not at scale. What we can tell you is what it costs to run the agents on top of it, which we measured separately: $0.0445 per task across nineteen tasks.


*ESR (esr.co) is built on these four services and audits its own error paths in public. Build a team free or read the changelog.*

Build your AI team

ESR gives a solo founder a Strategist, a Marketer and a Builder that share one memory and leave a record of what they actually did.

Build your team free

Keep reading

We built 297 things and never proved they workedOur append-only ledger has 963 items. 297 of them were built, shipped, plausible, and never once demonstrated to work. The number is not a backlog, it is a measurement of how much faster building got than proving.Twenty two seconds to a plan, measured three timesHow long it takes to go from a description of a business to a real AI generated plan, what we did when twenty two seconds felt broken, and the positive control that caught our own harness measuring nothing.