Circuit Breaker Pattern

Protect the caller when a dependency fails

A circuit breaker stops a caller from repeatedly attempting operations that are failing. It does not repair the dependency or guarantee that a fallback answer is valid. Use it with request deadlines, bounded concurrency and an explicit error contract.

The Azure pattern reference distinguishes a breaker from retry logic and describes the closed, open and half-open states.

Worked scenario: search suggestions

Assume a page can load without suggestions. A hypothetical suggestions service receives 100 calls/s; the caller allows ten concurrent requests and a 250 ms deadline. If every call times out, those ten slots can still be exhausted before a useful response returns. Retrying all failures immediately adds work to a service that is already struggling.

A breaker can stop new attempts after an observed failure threshold, allowing the main page to return a clearly unavailable suggestions section. An empty success response would hide the failure and pollute metrics. For a payment authorization dependency, the fallback must instead refuse or defer the action; inventing a successful authorization would break correctness.

Define each state precisely

A single successful request is weak evidence during a partial outage. A production library may support a rolling failure ratio, minimum call volume, slow-call thresholds and several probes. Choose these according to traffic and error semantics rather than copying one threshold into every service.

A small deterministic state model

This Python example models sequential admission and completion. It is deliberately not thread-safe, permits only one outstanding call, and does not wrap networking. Use a monotonic clock value as now; the caller must invoke complete once for an admitted request. A concurrent breaker needs synchronization and protection against late completions from an earlier state generation.

class Breaker:
    def __init__(self, threshold=3, cooldown=5):
        if threshold < 1 or cooldown <= 0:
            raise ValueError("positive threshold and cooldown required")
        self.threshold, self.cooldown = threshold, cooldown
        self.state, self.failures = "closed", 0
        self.reopen_at = 0
        self.in_flight = False

    def admit(self, now):
        if self.in_flight:
            return False
        if self.state == "open":
            if now < self.reopen_at:
                return False
            self.state = "half-open"
        self.in_flight = True
        return True

    def complete(self, succeeded, now):
        if not self.in_flight:
            raise RuntimeError("no admitted request")
        self.in_flight = False
        if succeeded:
            self.state, self.failures = "closed", 0
        else:
            self.failures += 1
            if self.state == "half-open" or self.failures >= self.threshold:
                self.state = "open"
                self.reopen_at = now + self.cooldown

breaker = Breaker()
for now in (0, 1, 2):
    assert breaker.admit(now)
    breaker.complete(False, now)
assert not breaker.admit(3)
assert breaker.admit(7)  # One recovery probe
assert not breaker.admit(7)
breaker.complete(True, 7)
assert breaker.state == "closed"

Coordinate retries and recovery

Keep retries inside one end-to-end time budget. Use jitter to avoid synchronized retry waves, and retry state-changing operations only with appropriate idempotency and outcome recovery. A breaker for each process may produce many simultaneous half-open probes across a fleet; local bounds do not automatically create a global bound.

Scope breakers by dependency and failure domain. One failing tenant or optional endpoint should not necessarily block all healthy operations. Count dependency failures separately from validation errors and caller cancellations.

Test the unhappy path

Force timeouts until the breaker opens. Verify that no new downstream requests start during cooldown, and that rejected calls have a distinct metric. Delay a successful call until after another call opens the breaker; it must not incorrectly close a newer failure generation. Fail a probe and confirm the circuit reopens. Finally restore the dependency and measure recovery without releasing the entire queued backlog at once.

Continue with backpressure, rate limiting and observability.