cascading failures
how one slow service takes down everything upstream, with the arithmetic.
“it was a 200ms timeout. we thought that was fine.”
how one slow service takes down everything upstream, with the arithmetic.
“it was a 200ms timeout. we thought that was fine.”
a cascading failure is one component's failure causing failures in its dependents, and theirs in turn, until a large part of the system is down. the initial fault is usually small and boring. the propagation is what turns it into an outage.
the mechanism is not mysterious and it is not bad luck. it follows from queueing theory, and you can predict the exact point at which it happens.
service A depends on B. under normal conditions A calls B, B answers quickly, A's worker threads are held briefly and released.
how many threads does A need? little's law answers it exactly:
L = λWconcurrency equals arrival rate times service time. A serving 500 requests per second with a 10ms downstream call needs 500 × 0.01 = 5 threads in flight. with 100 threads in its pool, A has enormous headroom.
now B slows down. B does not crash. it just gets slower. A's arrival rate has not changed, so the thread requirement rises linearly with B's latency:
B latency threads needed utilization served shed
10ms 5 5% 500rps 0rps
50ms 25 25% 500rps 0rps
100ms 50 50% 500rps 0rps
200ms 100 100% 500rps 0rps
500ms 250 250% 200rps 300rps
1000ms 500 500% 100rps 400rps
2000ms 1000 1000% 50rps 450rpsA's capacity limit is 100 threads ÷ 500 rps = 200ms of downstream latency. below that, nothing is visibly wrong. above it, A's pool is fully consumed, new requests queue, and A stops responding, while B is still up and still answering every request it receives.
25 / 10025%500 rps0 rps200ms (100 threads ÷ 500 rps). that number, not a crash, is the real capacity limit.the important part is the shape. this is not a gradual degradation that gives you time to react. A is completely healthy at 190ms and completely saturated at 210ms, because the resource runs out at a threshold rather than declining smoothly. and A's callers now see A as failed, so the same arithmetic starts one level up with A playing the part of B.
that saturation number is worth computing for your own services. it is a property of your thread pool and your traffic, it takes one division, and almost nobody knows theirs.
a crashed dependency fails fast. connection refused arrives in microseconds, the caller's thread is released immediately, error counters move, and circuit breakers open. the failure is loud and contained.
a slow dependency holds the connection. requests do not error. they wait, and waiting is exactly what consumes the caller's capacity. by the time anything detects a problem, the caller's pool is already gone.
this inverts the usual intuition about severity. the google SRE book chapter on cascading failures makes the same point from production experience: latency propagates upward through a system in a way that outright failure does not. a service that returns errors quickly is being a good citizen. a service that accepts your request and sits on it is spending your capacity without telling you.
the exhausted resource is usually one of four, and which one determines what you see.
threads are the common case, and the arithmetic above is the whole model. connection pools behave identically with a smaller and less visible limit, often a database pool of 20 rather than a thread pool of 200, so it saturates first. memory goes when queues are unbounded: a service that queues rather than rejecting will keep accepting work until the heap is gone, converting a latency problem into an OOM kill. file descriptors accumulate from sockets held open against a slow service, and the resulting EMFILE breaks everything in the process, including the parts that had nothing to do with the slow dependency.
unbounded queues deserve singling out. a bounded queue that rejects work is applying back-pressure. an unbounded one is just deferring the failure and making it worse, because every queued request is consuming memory to eventually be answered after the caller has already timed out. work that nobody is waiting for anymore is pure waste.
when requests start failing, clients retry. individually reasonable, collectively catastrophic, because the retry arrives exactly when the system has least capacity.
worse, it compounds through tiers:
retry amplification, 3 attempts per tier:
1 tier(s): 3x load at the bottom
2 tier(s): 9x load at the bottom
3 tier(s): 27x load at the bottom
4 tier(s): 81x load at the bottomthree tiers each retrying three times means the bottom service sees 27x its normal load at the moment it is already struggling. this is how a brief blip becomes a sustained outage: the system generates enough extra load to keep itself down long after the original fault has cleared.
the mitigations are well documented in the AWS builders' library on timeouts and retries. exponential backoff spreads attempts out. jitter stops clients synchronizing into waves. retry budgets cap retries as a fraction of total traffic, which is the one that actually bounds amplification. backoff without a budget still lets every client retry, just later. and retries should happen at one layer, not at every layer, or you get the multiplication above.
each defence maps onto a specific step in the mechanism above.
timeouts bound W in little's law, which bounds the thread requirement. this is the foundational one. an infinite timeout means unbounded concurrency demand. set it from the dependency's real p99 plus headroom, not from a round number.
circuit breakers stop sending traffic to a failing dependency, converting a slow failure into a fast one and freeing your capacity immediately. they also give the struggling service room to recover instead of holding it under.
bulkheads partition the resource. if calls to B draw from a dedicated pool of 20 threads, B's slowness can consume at most those 20, and traffic that does not touch B is unaffected. the failure is contained to the fraction of capacity you allocated to it.
load shedding and back-pressure reject work you cannot complete instead of queueing it. shedding load is counterintuitive, deliberately failing requests to stay up, but serving 70% of traffic beats queueing 100% of it into a collapse.
these are covered properly in the resilience chapter. what matters here is that they are not a grab-bag of best practices: each one bounds a specific term in an equation you can compute in advance.
that is the genuinely useful property of cascading failures. the conditions that produce them are unbounded timeouts, shared pools, unbounded queues, and unbudgeted retries, and every one of those is visible in a config file on a quiet afternoon, well before it turns into an incident.