Phi Accrual Failure Detector

Suspicion is not proof of failure

A fixed timeout asks whether the last heartbeat arrived too long ago. A phi accrual detector instead measures how surprising the current delay is under a model of recently observed heartbeat intervals. The application chooses what action to take when the suspicion score crosses its threshold.

This distinction matters during a network partition: a detector observes silence, not the remote process. It cannot calculate an unconditional probability that a node has died.

The score

Let $F(t)$ be the modeled cumulative distribution of heartbeat intervals and $t$ the elapsed time since the last heartbeat:

The tail probability $1-F(t)$ describes an interval at least this long under the model. A smaller tail probability produces a larger suspicion score.

These are not measured probabilities of a real node crash. A bad distribution fit, correlated network loss, or a process pause can produce a misleadingly large score.

Building the detector

1. Record inter-arrival intervals with a monotonic clock. Wall-clock corrections should not change elapsed time.
2. Keep a bounded history. Require enough samples before trusting the estimate.
3. Fit or approximate an interval distribution. A normal model is one choice, not a requirement of every accrual detector.
4. Apply a minimum variance and an acceptable heartbeat-pause allowance so a very stable history does not make a tiny delay appear catastrophic.
5. Compute the score and compare it with the configured threshold. Keep the score separate from membership and failover policy.

For example, two peers can have the same expected interval but different jitter. A 200 ms silence can be much more suspicious on a tightly clustered 100 ms heartbeat stream than on one with broad variation. The actual scores depend on the distribution and safeguards; there is no universal mapping from that silence to phi.

What adaptation can and cannot do

Recent interval history can accommodate gradual timing changes. An abrupt network pause may arrive before the history adapts, so a higher threshold trades detection speed against false suspicion. Akka documents a phi detector based on a normal distribution with tunable safeguards; do not assume every product named “phi accrual” uses identical mathematics or defaults.

A failure-injection exercise

Use three separate experiments: kill the sender, delay its heartbeat packets, and pause its process while keeping the host reachable. Record the score and recovery time in each case. The last two should remind you why an unreachable node is not necessarily a dead node.

Before promoting a replacement writer, use a fencing or consensus mechanism that prevents the old writer from continuing to modify shared state. Detection tells you when to investigate or reconfigure; it does not by itself make that reconfiguration safe.

Source and further reading

Akka: Phi Accrual Failure Detector documents the score and tuning parameters. Compare this approach with heartbeat timeouts and leader election.