partial failure
what makes distributed failure different from a crashed process.
what makes distributed failure different from a crashed process.
in a single process, failure is mostly binary. the function returned or it threw. the process is running or it is not. recovery is correspondingly simple: catch it, retry it, restart it.
distributed systems removed that guarantee and did not replace it with anything. a partial failure is some components working while others do not, with the system as a whole still running in a state nobody designed. essentially every hard distributed systems problem is downstream of this.
a replica falls behind and keeps serving reads. queries return data from four minutes ago. nothing errors.
a service holds its listening socket but its downstream connection pool is exhausted. some requests complete instantly from cache; the rest block until timeout. the p50 looks fine and the p99 is a cliff.
an availability zone loses connectivity to another. half your nodes can reach the cache and half cannot, so half start hammering the origin, and the half that are fine make the aggregate dashboards look fine too.
a disk is failing but has not failed. writes succeed. reads from most sectors succeed. reads from a few hang for seconds and then error.
in every case the system is partially working, no alarm is firing, and anything watching only the success path sees nothing wrong.
send a request, get no response. four things could have happened:
the request never arrived. it arrived and was not processed. it was processed and the response was lost. it is still being processed right now.
these are indistinguishable from outside, and they demand opposite responses. retrying is correct for the first two and potentially a duplicate charge for the third and fourth. you cannot resolve the ambiguity by waiting longer, because "slow" and "never" look identical until you give up. this is the same asynchronous-model limitation that makes failure detection unsolvable in general, and it is why idempotency is not a nice-to-have but the only thing that makes retries safe.
thinking of services as up or down is what makes partial failure surprising. the useful model has states in between, and the important column is what your monitoring reports for each.
healthydegradedoverloadedunresponsivecrashedthe two states a health check describes honestly are the two ends. everything in the middle, where most real incidents live, returns 200 OK while the service is already failing its callers.
that gap explains a familiar incident shape: the dashboards are green, the on-call engineer cannot find anything wrong, and users are complaining. nothing is broken in the way the monitoring was built to detect.
stop treating "no error" as success. the operation may be queued, lost, or still running. if the outcome matters, confirm it explicitly rather than inferring it from the absence of an exception.
make operations idempotent. since you cannot know whether a request was processed, you need to be able to send it again safely. an idempotency key on writes turns an unanswerable question into a cheap one.
detect from the caller. a degraded service frequently cannot detect its own degradation. that is what makes it degraded. callers see the timeouts, the error rates, and the latency shift. circuit breakers work precisely because they measure real request outcomes instead of asking the service how it feels.
degrade on purpose. decide in advance what your service does with a half-broken dependency. serve stale data, drop the optional feature, return a partial response. a designed degraded mode is the difference between a partial failure and an outage.
none of this eliminates partial failure. it is not a defect to be fixed. it is a property of running software on multiple machines connected by a network that can lose messages. the systems that handle it well are the ones that assumed it from the start.