~ $ cat ./notes/uptime-payments-backend.md

Keeping 99.9% uptime on a payments backend

July 2026 · 6 min read

99.9% uptime sounds like an infrastructure metric. It isn't — it's a user metric. It means that out of every thousand times someone tries to pay, redeem a gift card, or load a balance, at most one of those attempts is allowed to fail for a reason that's our fault. Everything below follows from taking that framing seriously.

None of these patterns are clever. That's the point. On a payments backend, clever is a liability; the systems that stay up are the ones where every failure mode was decided on purpose, in advance, by someone who wrote it down.

1. Every network call gets a timeout — chosen, not defaulted

The default timeout of most HTTP clients is either infinity or something absurd like two minutes. On a payment path, a request that hangs for two minutes is worse than one that fails in two seconds — it holds a connection, a worker, and a user, and then probably fails anyway.

// Budget the whole operation, then divide it up.
// User-facing SLA: respond in 3s.
const PARTNER_TIMEOUT = 1200;   // slowest partner call
const DB_TIMEOUT = 500;         // primary store
const CACHE_TIMEOUT = 100;      // cache is optional — fail open

// If the budget doesn't fit, the design is wrong,
// not the timeout.

The discipline that matters: timeouts are derived from the user-facing budget, top-down. If the pieces don't fit inside the budget, you change the design (cache it, precompute it, make it async) — you don't quietly raise the timeout.

2. Retries with jitter — and only on idempotent operations

Retries are how transient failures become invisible, and also how outages become worse. A fleet of clients retrying in lockstep after a blip is a synchronized wave that hits your recovering service like a second outage. Jitter breaks the synchronization.

async function withRetry(fn, { attempts = 3, base = 200 } = {}) {
  for (let i = 0; i < attempts; i++) {
    try {
      return await fn();
    } catch (err) {
      if (!isTransient(err) || i === attempts - 1) throw err;
      // full jitter: sleep anywhere in [0, base * 2^i)
      const cap = base * 2 ** i;
      await sleep(Math.random() * cap);
    }
  }
}

The rule that keeps this safe: reads retry freely; writes retry only when the operation is idempotent. Which brings us to the pattern that makes writes retryable at all.

3. Idempotency keys: make retries indistinguishable from single calls

Partner networks time out and retry. Mobile apps resend on flaky connections. If a redemption request arrives twice, the system must move money exactly once. The client sends a unique key per logical operation; the server remembers the outcome per key.

// First call with key "tx-91af": executes, stores result.
// Any later call with "tx-91af": returns the stored result.
app.post("/redeem", async (req, res) => {
  const key = req.headers["idempotency-key"];
  const cached = await outcomes.get(key);
  if (cached) return res.status(cached.status).json(cached.body);

  const result = await redeem(req.body);       // moves money
  await outcomes.set(key, result, TTL_24H);    // remember it
  res.status(result.status).json(result.body);
});

Once this exists, an entire class of incident disappears: the timeout-then-retry double-spend simply cannot happen, no matter how badly the network behaves.

4. Fail fast, fail partial, never fail totally

• ▸Cache down? Serve from the database, slower. A cache miss must never be an error.
• ▸One domain module degraded? The other domains keep serving — isolation boundaries exist precisely for this moment.
• ▸A dependency flapping? Stop calling it for a while (circuit breaking) instead of queueing doomed requests behind it.

The shared idea: decide in code what the system does when each dependency is gone. If you haven't decided, the answer at 3 a.m. is 'everything breaks at once'.

5. Monitor what users feel, alert on symptoms

CPU graphs don't page anyone at a good shop. The signals that map to the 99.9% promise are p99 latency and error rate per endpoint — those are what a user experiences. Everything else (memory, connections, queue depth) is diagnosis, not detection.

One habit worth stealing: every alert should complete the sentence 'a user is currently unable to ___'. If it can't, it's a dashboard, not an alert.

6. Deploys are where uptime goes to die

Most self-inflicted downtime ships on a Friday inside a big release. The countermeasures are process, not technology: small changes, one at a time, quick to roll back, behind flags when risky. A deploy you can revert in one minute is a deploy you can make boldly.

That's the whole list. Nothing here is publishable research — it's just the discipline of deciding failure behavior on purpose. Systems should be boring in production and interesting in the design doc.