Heartbeat Protocol

The Problem

In distributed systems, servers crash. Networks fail. How do you know if a server is still alive or has failed?

Solution: The Heartbeat Protocol.

• The Beat: Every T seconds (e.g., 1s), Server A sends a "Heartbeat" signal to a monitor.
• The Timeout: If the monitor doesn't receive a signal for a threshold (e.g., 3 consecutive beats), it marks Server A as suspected or unreachable. A delayed heartbeat cannot distinguish a crash from a network partition.

Implementation Strategies

Push Model

Server actively sends heartbeats to monitor.

import time
import threading

class HeartbeatSender:
    def __init__(self, server_id, interval=1.0):
        self.server_id = server_id
        self.interval = interval
        self.running = False
    
    def start(self):
        self.running = True
        threading.Thread(target=self._send_loop, daemon=True).start()
    
    def _send_loop(self):
        while self.running:
            print(f"[{self.server_id}] Heartbeat sent")
            time.sleep(self.interval)

Pull Model

Monitor actively pings servers.

class HeartbeatMonitor {
    async checkServer(serverUrl) {
        try {
            const response = await fetch(`${serverUrl}/health`, {
                signal: AbortSignal.timeout(5000)
            });
            return response.ok;
        } catch {
            return false; // Unreachable within this timeout; not proof of a crash
        }
    }
}

Advanced: Adaptive Timeout

import statistics

class AdaptiveMonitor:
    def __init__(self):
        self.latencies = []
    
    def calculate_timeout(self):
        if len(self.latencies) < 5:
            return 3.0
        
        mean = statistics.mean(self.latencies)
        stddev = statistics.stdev(self.latencies)
        return mean + (3 * stddev)  # Heuristic only; real heartbeat intervals may not be normally distributed

Real-World Examples

Kubernetes

• Node status and Lease objects provide heartbeat signals.
• Detection and eviction are separate controller decisions.
• Timings depend on cluster configuration and version; inspect the configured lease, grace period, and tolerations instead of copying a universal timeout.

Redis Sentinel

• Pings master every 1s
• Configured down-after-milliseconds timeout → Subjectively Down
• Quorum agreement → Objectively Down
• Triggers automatic failover

Cassandra

• Uses Phi Accrual Failure Detector
• Gossip protocol every 1s
• Phi threshold: 8 (suspicion level)
• Adaptive to network conditions

Common Patterns

Flapping Detection

Require N consecutive failures before marking a node suspected. Reset the count on a successful observation:

class FlappingDetector:
    def __init__(self, required_failures=3):
        self.consecutive_failures = 0
    
    def heartbeat_missed(self):
        self.consecutive_failures += 1
        return self.consecutive_failures >= self.required_failures

    def heartbeat_received(self):
        self.consecutive_failures = 0

Key Metrics

• Heartbeat success rate
• Average latency
• Timeout frequency
• False positive rate

Interview Tips

• Timeout tradeoff: Short = fast detection, more false positives. Long = fewer false positives, slower detection.
• Adaptive timeouts: A mean-plus-margin rule is a simple heuristic; an accrual detector exposes a suspicion score rather than proof of failure.
• Network partitions: Require quorum to avoid split-brain scenarios
• Push vs Pull: Push scales better for many servers; Pull better for dashboards
• Recovery test: Delay heartbeats without killing the process. Verify that failover fences the old writer before accepting a replacement.

Related Concepts

• Leader Election
• Raft Consensus
• Gossip Protocol
• Circuit Breaker

Sources and verification

Kubernetes node heartbeats describes node status and leases. Redis Sentinel distinguishes subjective from objective failure detection. Neither a timeout nor a quorum observation by itself proves that a remote process has stopped.