Timeouts That Don't Cancel: Why a Client Abort With No Server Timeout Is a Load Multiplier

An 8-second client-side abort on every database call felt like a safety net. With no statement_timeout on Postgres it was the opposite: every timed-out query kept running while the client retried on top of it, so under saturation load could only add, never shed. A postmortem of a Postgres death spiral — the chronic base load nobody read, the bulk onboarding that lit it, the retry loops that fed it, and the one server-side setting that breaks the cycle.

TL;DR — Every database call in the API carried a client-side abort at 8 seconds. The database had no statement_timeout. That combination means a timeout does not cancel anything: the client gives up, the query keeps burning CPU on the server, and the client retries — stacking a new copy of a query whose predecessor is still running. On a single Postgres instance that had been running at the edge of saturation for a week, a bulk onboarding of ~800 users pushed it over, and from that point load could only add, never shed. Every query timed out, including single-row primary-key lookups. The fix that breaks the cycle is one line on the server, and the lesson is that a timeout is only a safety net if both ends honour it.

Key takeaways

A client-side abort frees the client. Unless something on the server side cancels the statement, the query runs to completion for nobody, and a retry starts a second copy.
Under saturation, “timeout then retry” is a load multiplier. Each retry adds work; nothing removes any. The system cannot recover on its own because recovery requires load to fall.
A server-side statement_timeout bounds the damage of one slow query to a few seconds of one connection. It is the single cheapest change that turns a death spiral into a slow afternoon.
A memory-based process restart under load is a herd generator: it drops every long-lived connection at once and they all come back at once.
The saturation was in the logs six days before the outage — 120 client-side aborts, a p99 of 2.7 s, single-row lookups taking longer than 8 s. Nobody was reading them.

Four layers, none fatal alone

  1. chronic base load          p99 2.7 s, 120 aborts/day, for at least a week
                │
  2. trigger                    ~800 users bulk-onboarded straight into production
                │
  3. amplifier                  client aborts at 8 s, server never cancels, clients retry
                │
  4. structural collapse        OOM restart → every stream reconnects at once → repeat
                ▼
        CPU and RAM pinned, every query times out, the DB dashboard is unreachable

The setting: a food-tech SaaS on a single managed Postgres instance (2 GB) behind PostgREST, with a Node/Express API running as one PM2 process, doing roughly 100,000 requests a day. The incident is real and the numbers below come from the production log and the migration headers written during remediation. What I do not have is a clean “after” — the remediation was verified by the timeouts stopping, not by a before/after benchmark, and I will say which fixes were measured and which were not.

The layers matter because each one has an obvious local fix, and fixing any one of them in isolation would have produced a system that failed the same way a few weeks later from a different trigger. The one that generalises is the third.

The week before: the log already said so

Six days before the incident, a 24-hour slice of the production log looked like this:

Metric	Value
Requests / 24 h	~97,500
p50 latency	56 ms
p90 latency	192 ms
p99 latency	2,697 ms
Max latency	20,222 ms
Client-side `TimeoutError` aborts	120

The p50 is healthy, which is why nobody looked. The p99 is the number that was screaming, and the 120 aborts are the number that should have paged. Some of those aborts were on single-row lookups by primary key — a query that takes under a millisecond on a healthy database. A single-row indexed lookup exceeding 8 seconds does not mean the query is slow. It means the server is saturated and the query is waiting its turn.

That distinction is the whole diagnosis, and it was available a week early. I wrote about a different failure that sat in the logs for four days because nothing was watching the one signal that was unambiguous; this is the same mistake with a different signal. A count of client-side aborts is a saturation meter. It should have had an alert on it at any number above zero.

Where the base load came from is a list of things I have written about before and will not re-explain here — a foreign-key column with no index on the largest table, a stored function that materialised an outlet’s entire order history before any filter applied, an N+1 that ran two extra queries per row of a paginated list, a subsidy quote that summed an entire ledger table in JavaScript on every checkout. The Postgres scaling post covers the techniques; mistakes #1, #4 and #12 in the retrospective cover the habits. What is relevant here is that the database was already spending most of its capacity on avoidable work, so there was almost no headroom when the trigger arrived.

The trigger: ~800 people and a script

A large corporate client went live. Two things happened the same day.

First, an onboarding script ran against production, using an admin credential that bypasses row-level security, to create ~800 user accounts. The arithmetic of that script is worth spelling out because “insert 800 rows” sounds trivial:

~2,400 client calls, roughly 6,000 SQL statements once you count what the API layer does per call.
800 calls to the auth service’s admin create-user endpoint, each of which is four to six statements on its own.
18 full sequential scans of the largest table, because the script checked for an existing account by email and phone number and neither column was indexed.
A retry wrapper that re-ran failures three times against the already-slow database.
A “does this user exist” check implemented by calling an endpoint that generates a login link — a write, used as a read, because it was the one call that returned a clean yes/no.

Second, those 800 people opened the app. Every order-history request for a new user was a full scan of the orders table (the missing foreign-key index). And the client’s admin dashboard called an order-list endpoint that sent every employee id in the URL — an in.(...) filter that had grown to about 30 KB of UUIDs and had started returning 400 at roughly 400 employees. The dashboard did not know that; it kept asking.

None of this is unusual. A bulk job on a healthy database would have been slow and finished. What made it an outage was what happened to every request that hit the 8-second wall.

The amplifier: abort is not cancel

Every call to the database went through a fetch wrapper that looked like this:

// Illustrative. This shape is everywhere: "never let a hung request block the API."
const db = createClient(url, key, {
  global: {
    fetch: (input, init = {}) =>
      fetch(input, { ...init, signal: AbortSignal.timeout(8_000) }),
  },
});

AbortSignal.timeout does exactly one thing: after 8 seconds it rejects the promise and closes the client side of the HTTP connection. It says nothing to Postgres. In our stack the query kept executing after the client was gone, and you should assume the same until you have proven otherwise: a Postgres backend only notices a disconnected client when it next tries to write to the socket, and a query that is still scanning or hash-joining has nothing to write yet. (Postgres 14 added client_connection_check_interval to poll for exactly this, and it is off by default.)

So the sequence on a saturated server is:

  t=0    client sends query Q1                      server: Q1 running
  t=8    client aborts, retries → Q2                server: Q1 running, Q2 queued
  t=16   client aborts, retries → Q3                server: Q1 running, Q2, Q3 queued
  t=24   client aborts, retries → Q4                server: Q1 finishing, Q2, Q3, Q4 …
                                                    nobody is waiting for any of them

The client experiences a timeout every 8 seconds. The server experiences a client that submits the same query every 8 seconds and never stops. Multiply by every client, and load becomes strictly additive — each retry stacks a new copy of a query whose predecessor is still running, and nothing in the system ever removes work.

Three retry loops were feeding this, and each one was individually reasonable:

A payment-status poller in the app, every 2.5 seconds, with the retry flag set even in the catch block — so a timeout was treated the same as “not yet paid”, and the loop continued at full speed.
A server-sent-events stream for live orders, with retry: 10000 in the response — a reconnect every 10 seconds after any drop, and NGINX’s proxy_read_timeout of 60 s was dropping idle streams about 3,000 times a day on a normal day.
App-side pollers on their own timers, none of which backed off on failure.

Not one of these loops had a budget. Not one of them slowed down when the thing it was polling slowed down. And the thing they were polling had no way to say no.

This is the part I want to be precise about, because “add timeouts and retries” is standard advice — I gave it myself, in mistake #13 of the retrospective, about third-party APIs. The advice is right for a dependency you do not control. Pointed at your own database, a timeout that the server does not honour is not resilience. It is a client that has been taught to give up on the response while insisting on the work.

The collapse: a memory restart is a herd generator

The API ran as a single PM2 fork with max_memory_restart set to 512 MB. Under normal load that setting never fires. Under this load, two things were growing in the Node process: a log line per request that included the full request body (~42 MB a day, all of it on the hot path), and realtime channels that were never freed.

The channel leak is a good example of a bug that only matters in a storm. Each SSE client got its own subscription to a change feed, named with a timestamp so that names were unique:

// Illustrative. One channel per connected client, torn down on close.
app.get('/orders/live', (req, res) => {
  const name = `orders-${outletId}-${Date.now()}`;
  db.channel(name)
    .on('postgres_changes', { event: '*', table: 'orders' }, push)
    .subscribe();

  res.write('retry: 10000\n\n');

  req.on('close', () => {
    // BROKEN. In the client version we ran, this constructs a *second* channel
    // handle with the same name and unsubscribes that one. The channel that is
    // actually receiving events is never referenced again and leaks for the
    // life of the process.
    db.channel(name).unsubscribe();
  });
});

// FIXED. Keep the handle you subscribed with; tear down by reference.
const channel = db.channel(name).on(/* … */).subscribe();
req.on('close', () => db.removeChannel(channel));

Now put the pieces together. Memory climbs. PM2 kills and restarts the process. Every open SSE stream and every realtime channel dies at once. Every client reconnects at once — and each reconnect creates a brand new channel, because the name includes Date.now(), so the process comes back and immediately re-subscribes N times, on a database that is already saturated, while the leaked channels from before the restart are gone but the queries they started are not. The next restart comes sooner.

A process restart on memory pressure is a sensible last resort for a leak you have not found. As a response to load it is exactly backwards: it converts the slow-but-connected state into a thundering herd, on a schedule. The Socket.IO write-up has the client-side half of this — every reconnect must resync from the authoritative store, which is another query per client per herd — and the zero-downtime deploy post has the process model that at least spreads the restart across workers.

Postgres, meanwhile, had its RAM going to hash-joining the entire orders table for requests nobody was waiting for, and the instance had swapped out 483 MB of its 2 GB. Even the managed-database dashboard, which queries the same instance, stopped loading. That is the moment you find out how much of your observability shares a fate with the thing it observes.

Four classes, not four bugs

The postmortem grouped everything found into four classes, because fixing the instances without naming the classes is how you have the same incident twice.

A. Missing indexes. A foreign-key column with no index on the orders table — Postgres does not create indexes for foreign keys automatically, only for primary keys and unique constraints. Email and phone unindexed on the largest table. A transactions table with zero secondary indexes. A column the code assumed unique with no unique constraint to make that true.

B. Unbounded and N+1 patterns. The order-history function that materialised everything before filtering, called from 14 places including the SSE handlers — which re-ran it per connected client per order update. Fetch-then-count-in-JavaScript in four places. Uncapped Promise.all loops doing two queries and two external calls per order. An hourly scheduler doing a thousand round trips per tick.

C. Additive-load timeout and retry design. This post.

D. Operational gaps. A bulk job run against production with an admin credential. A deployed VM running cron code that was not in any commit. Migrations applied by hand with no runner, so “code deployed, function missing” was a real failure mode. Cron jobs registered by import side effect with no overlap locks. Request-body logging on the hot path.

Classes A and B are the ones every scaling post covers, including mine. They were the fuel. Class D was the match. Class C is why the fire could not go out.

What breaks the spiral: a deadline the server enforces

The change that removes the multiplier is one statement per role:

-- Illustrative. Per-role, so the API's roles get a tight bound and the
-- admin role used by maintenance jobs gets a looser one.
ALTER ROLE app_user    SET statement_timeout = '10s';
ALTER ROLE app_anon    SET statement_timeout = '10s';
ALTER ROLE app_admin   SET statement_timeout = '30s';

statement_timeout aborts any statement that runs longer than the limit, from the server’s own clock, whether or not anyone is still listening. With it in place, a slow query costs at most 10 seconds of one connection instead of an unbounded number of stacked copies. The retry storm still happens — the clients have not changed — but every copy it creates is now guaranteed to die, so load can shed. That is the difference between a spiral and a spike.

Three rules follow from it:

The client abort must be longer than the server timeout, not shorter. If the client gives up at 8 s and the server at 10 s, the last two seconds of every slow query are orphaned work again. Set the server bound first; set the client bound above it so the client normally sees the server’s error rather than its own.
Never retry unconditionally from a catch block. A timeout is a signal that the server is busy, and the correct response to “busy” is to wait longer, not to ask again sooner. Exponential backoff with jitter, and a budget after which the poller gives up and shows the user a “check again” button.
A per-role setting is a deliberate choice. The API roles get the tight bound because nothing they do should take 10 seconds. The admin role gets 30 s because the maintenance jobs that use it legitimately do bigger work — and if they need more than that, they should SET LOCAL statement_timeout for the one statement that needs it, in a transaction, so the exception is visible in the code.

Honest status: the role-level timeout went in after the seq scans and the retry loops were dealt with, not before. That ordering was deliberate — the plan was to remove the baseline first so that the timeout, once it landed, would fire on genuine anomalies rather than on every busy lunch — and I think it was right. What I do not have is a measured “after” for the timeout itself: it was applied as a guard against the next incident, not tuned against this one, and the next incident is the only thing that will tell me whether 10 seconds was the right number.

What was fixed, and what the evidence was

The index diagnosis came from pg_stat_user_tables, and the headline number is one I still find hard to believe: a 114-row table had been sequentially scanned 1.6 billion times and had 166 billion tuples read from it. The tables were small, so each scan was cheap; the cost was the repetition — correlated subqueries and per-row embeds executing once per outer row — and re-reading those heap pages billions of times is what kept the instance in constant memory pressure. That mechanism gets its own short post; the fix was CREATE INDEX CONCURRENTLY on seven tables, run outside a transaction, during service.

The stored function was converted from PL/pgSQL to LANGUAGE sql STABLE so that the planner can inline it and PostgREST’s filters push into the plan instead of applying after the function has materialised the outlet’s lifetime history. One function change; 14 call sites fixed at once.

The 30 KB URL became a join inside Postgres — a function that takes the organisation id and does the join, filter, sort, pagination and count in SQL. For the sites that were not worth a function, a small helper splits an in.(...) filter into chunks of 100 ids and runs them with bounded concurrency. One design note from that helper is worth a paragraph: the query builder is single-use (awaiting it sends the request), so the helper cannot take a query and re-run it per chunk — it takes a factory that builds a fresh query per chunk. And it refuses queries that use count, .single(), .order() or .range(), because those are per-request semantics that cannot be stitched back together across chunks. A helper that silently returned the wrong count would have been worse than the 400.

Aggregations moved into SQL: one function returning six counts in one row, backed by a new composite index, instead of fetching every open order row and counting in JavaScript. A function with an accidental cross join — a LEFT JOIN with no join predicate — was rewritten and verified row-for-row identical against the old one with EXCEPT in both directions on a local Postgres 17 with seeded edge cases.

Every rewritten endpoint was diffed field by field against the old response, and the quirks were preserved on purpose (an absent key rather than 0 for an empty bucket, because a client somewhere checks for the key). Three edge-case differences were documented rather than hidden, and one of them was a latent bug in the old code: a .single() on a whitelist row that erred when the row was duplicated, returned false, and hid an outlet from a user who was whitelisted twice. The batch version returns true. Nobody had reported it.

What was measured: the client-side abort count in the log went to zero after the index and function changes, and the dashboard came back. What was not measured: a before/after on p99, CPU, or seq-scan counts. I would have liked to end this post with those numbers and I do not have them. The remediation plan’s last line was to reassess the compute tier only after the fixes — on the theory that a 2 GB instance is fine once it stops reading a 114-row table a billion times — and that decision has held so far.

The bulk-onboard runbook that should have existed

Written after the fact, in the order I wish I had read it before:

Batch the inserts — one statement per hundred rows, not one per row.
Throttle the one step that is genuinely per-row (the auth service’s admin call) and pace between waves. A few hundred accounts an hour was fine; a few hundred a minute was not.
Run it off-peak. The lunch window is 80% of the day’s orders in 90 minutes; the go-live was in it.
Never reuse the API’s client for a script. The 8-second abort that is reasonable for a user-facing request is a retry generator for a batch job. Scripts get their own client with a long timeout and no automatic retry.
Never point a script at production with an admin credential. Stage it, or run it through the API with a normal role and let the API’s guards say no.

What I got wrong beyond the bug

I had the warning a week early and did not read it. 120 aborts and a 2.7-second p99 in a log that was being written every day and queried by nobody. The saturation meter existed; the alert did not.

I thought an 8-second abort was a safety net. It felt like defensive engineering: never let a hung call block a request. Without a server-side counterpart, it was a mechanism for converting slow requests into abandoned ones and abandoned ones into duplicates. The intention was load shedding. The effect was load generation.

Retrying from a catch block forever is the same bug in a different file. The poller that treated a timeout as “not paid yet” was written by someone who assumed timeouts were rare, and that someone was me.

I let a restart-on-memory setting stand in for finding the leak. max_memory_restart is a tourniquet. Leaving it on for months meant that when the leak finally mattered, the tourniquet was the thing generating the herd.

FAQ

Does a client-side timeout cancel the query on Postgres? No. Aborting a fetch or closing a connection frees the client; the Postgres backend keeps executing the statement until it finishes or until it next tries to write to the closed socket. To bound work on the server, set statement_timeout (per role, per database or per session), and on Postgres 14+ consider client_connection_check_interval so long-running queries notice a vanished client sooner.

What is a retry storm and why does it happen under database overload? A retry storm is when clients respond to slow or failed requests by re-sending them, so the number of in-flight requests grows exactly when the server has the least capacity to serve them. Under overload, every timeout produces a retry, every retry adds load, and nothing removes any — load becomes additive and the system cannot recover until something external sheds work.

Should the client timeout be longer or shorter than statement_timeout? Longer. If the client gives up before the server does, the tail end of every slow query runs for nobody. Set the server’s statement_timeout as the real deadline and the client’s abort slightly above it, so the client normally receives the server’s cancellation error and never orphans work.

What should statement_timeout be set to in production? A value slightly above the slowest query you consider legitimate for that role. For a request-serving API role, single-digit seconds is typical; maintenance and reporting roles get a looser bound, and individual statements that need more can SET LOCAL statement_timeout inside a transaction so the exception is explicit in code.

Why does a process restart cause a thundering herd? A restart drops every long-lived connection — websockets, server-sent-event streams, change feeds — at the same instant, and every client’s reconnect logic fires at the same instant. Each reconnect typically re-subscribes and re-fetches state, so the process comes back to a burst of load larger than the one that caused the restart. Restart on memory pressure only as a last resort, and never as the primary response to load.

How do you detect database saturation before it becomes an outage? Alert on the count of client-side timeouts, on p99 latency rather than p50, and on any indexed single-row lookup exceeding a few hundred milliseconds — a primary-key lookup that takes seconds means the server is queueing, not that the query is slow. Those signals were present six days before this incident.

Why did a small table need an index if a full scan is cheap? Because the scan ran once per outer row of a larger query. A 114-row table scanned 1.6 billion times reads 166 billion tuples, and re-reading the same pages that often evicts useful data from shared buffers. Per-scan cost is irrelevant when the multiplier is the number of rows in the driving query.

What I’d still improve

Measure the role-level statement_timeout. It is applied; what is missing is a week of p99 and abort counts with it in place, and a count of how often it actually fires, so the number is defended by data rather than by the postmortem’s reasoning.
One shared change feed per outlet, fanned out to its SSE clients in-process, instead of one channel per connection. The leak fix stops the bleeding; this removes the artery.
A request deadline at the Express layer, propagated as the remaining budget into every downstream call, so a request that has already taken 9 seconds does not start a 10-second query.
Replace restart-on-memory with cluster mode and a drain, so a worker that must recycle does it alone and its connections move to a sibling instead of reconnecting en masse.
Alert on TimeoutError count and p99 > 1 s. Two rules on a log that already exists.
Turn on pg_stat_statements.track_planning. A later incident on the same database — a trigger that re-planned its upsert on every call — hid from the statistics view behind exactly that default.
A migration runner and a cron registry, so “code deployed, function missing” and “the VM runs something not in git” stop being possible states.

The one idea to take away

A timeout is only a safety net if both ends honour it.

A client-side abort with no server-side counterpart does not shed load; it converts load — from work someone is waiting for into work nobody is waiting for, plus a retry. On a healthy server that conversion is invisible. On a saturated one it is the mechanism that keeps the server saturated, and no amount of index tuning will end an incident whose load can only go up. Put the deadline where the work happens, make the client’s deadline longer than the server’s, and make every retry loop slow down when the thing it is calling does.

I write about backend reliability, Postgres data modelling and production failure modes from work on food-tech and healthcare platforms. More on what I build and how I work, the techniques behind the index and function fixes, and why the warning sat unread in a log for a week. If your API has a client timeout on every database call, go and check whether the database has one too.