failure detection

failure detection is deciding whether a remote component is working. it sounds like an engineering problem with a correct answer. it is not. there is a proof.

the impossibility result

in an asynchronous system, where messages can be delayed arbitrarily and clocks are not synchronized, a crashed node and an arbitrarily slow node are indistinguishable. no amount of waiting resolves it, because "has not answered yet" is a state both of them occupy.

fischer, lynch and paterson made this precise in 1985: impossibility of distributed consensus with one faulty process proves that no deterministic algorithm can guarantee consensus in an asynchronous system if even one process may fail. chandra and toueg later showed what it takes to escape. consensus becomes solvable with an unreliable failure detector that is allowed to be wrong, provided it is eventually accurate.

this is not academic trivia. it is the reason every mechanism below is a heuristic with a tuning knob rather than an answer. all of them trade false positives (declaring a healthy node dead) against detection latency, and no setting makes both zero.

timeouts

the primitive form. if no response arrives within some period, call it failed.

the entire difficulty is the number. too short and healthy requests are declared failures, which triggers unnecessary failovers and, if you retry, adds load to a system that was fine. too long and threads sit blocked, detection lags, and the cascade forms before you react.

the value is not derivable from first principles. it comes from the dependency's measured latency distribution plus headroom: a service with a p99 of 200ms might get a 500ms timeout, and one that occasionally does 2 seconds of legitimate work needs more, accepting slower detection as the price.

what a timeout does not give you is a cause. crashed, overloaded, partitioned, or briefly slow all produce the same expired deadline. you learn that a deadline passed and nothing else.

heartbeats

a periodic "still here" message. miss enough of them and the sender is declared dead.

the tuning is interval against certainty. a 1-second heartbeat with a 3-miss threshold detects in roughly 3 seconds; a 10-second heartbeat with the same threshold takes 30. faster means more traffic and more false positives, because the things that delay a heartbeat, a GC pause, a scheduling hiccup, a brief network stall, are common at short timescales.

binary thresholds handle this badly. one late heartbeat from a node that is garbage collecting is not evidence of death, but a hard threshold cannot express "probably fine".

phi accrual detection replaces the boolean with a continuous suspicion score. the φ accrual failure detector (hayashibara, défago, yared and katayama, SRDS 2004) models the distribution of observed inter-arrival times and outputs

φ = −log₁₀ P(heartbeat arrives later than the current gap)

so φ rises smoothly as a heartbeat gets later relative to how late heartbeats usually are on this link. φ = 8 means the probability of being wrong to suspect is roughly 10⁻⁸. because it is derived from observed history, it adapts on its own: a link that is normally jittery raises the bar before suspecting, while a normally punctual one reacts fast.

cassandra ships this with phi_convict_threshold defaulting to 8 over a window of recent arrivals, and the documented advice for flaky networks, notably cloud environments, is to raise it to around 12 rather than tighten it. akka implements the same detector and documents its own threshold tuning. the win is that callers choose their own threshold from one shared signal: a cheap cache read can act on φ = 5 while a leader election waits for φ = 12.

health checks

an endpoint the service exposes about itself, polled by load balancers and orchestrators.

they are worth exactly as much as what they check. an endpoint that returns 200 OK from a static handler is worse than nothing, because it converts an unknown into a confident wrong answer. one that exercises real dependencies reports something meaningful.

deep checks have their own failure mode, though. if every instance checks the database every few seconds, the health checking itself becomes load, and during a database slowdown, every instance fails its check simultaneously and the orchestrator pulls the entire fleet. the check turned a degraded dependency into a total outage. shallow checks are safe to run often; deep checks should be rate-limited, cached briefly, and should not fail the whole instance for a non-critical dependency.

kubernetes splits the concern into three probes, and the split is the useful idea:

livenessProbe:      # is the process wedged? failing → restart the container
  httpGet: { path: /healthz, port: 8080 }
  periodSeconds: 10       # default
  failureThreshold: 3     # default
  timeoutSeconds: 1       # default

readinessProbe:     # can it serve right now? failing → pull from the LB, do not restart
  httpGet: { path: /ready, port: 8080 }

startupProbe:       # still booting? suppresses the other two until it passes
  httpGet: { path: /healthz, port: 8080 }
  failureThreshold: 30
  periodSeconds: 10

the documented defaults mean a liveness failure takes up to about 30 seconds to act on: three periods of 10 seconds. worth knowing before you assume detection is instant.

the distinction that saves outages is liveness versus readiness. a service waiting on a temporarily unavailable dependency should fail readiness. stop receiving traffic, stay alive, recover when the dependency does. if that is wired to liveness instead, kubernetes restarts every instance, and now you have a cold fleet with empty connection pools all trying to reconnect to the dependency that was already struggling. this is a common way to convert a recoverable dependency blip into a self-inflicted outage.

startupProbe exists because slow-starting applications otherwise force you to set a liveness threshold generous enough to cover boot time, which then makes real hangs take minutes to detect. it lets boot and steady state have different budgets.

what none of it catches

detection latency is irreducible. there is always a window between failure and detection, and traffic goes into that window. you can shrink it by detecting faster, which raises false positives. this is the tradeoff curve from FLP, and every system is picking a point on it whether deliberately or by leaving defaults alone.

a check only sees what it checks. a 200 OK says the health endpoint worked. it says nothing about a service that is degraded, serving stale data, or failing for one tenant. those are the partial failures that health checks structurally cannot see.

which is why detection alone is not a strategy. circuit breakers judge by real request outcomes rather than a synthetic endpoint. latency tracking catches services that pass every check and are unusable. distributed tracing catches failures confined to one code path. these observe actual traffic, and actual traffic is the only thing that exercises the paths users are on.

detection tells you when something has stopped. the failures that cost the most are the ones where it did not stop. it just started being wrong.