Circuit Breaker, Bulkhead and Retry
Resilience patterns protect finite capacity, but their interaction can amplify load when applied independently.
Circuit Breaker, Bulkhead and Retry
Resilience patterns protect finite capacity, but their interaction can amplify load when applied independently. A timeout bounds waiting; a retry repeats work; a breaker suppresses calls to an unhealthy dependency; a bulkhead isolates resource pools. Breakers use closed, open and half-open states. Retries need bounded exponential backoff with jitter, a shared deadline and an idempotent operation.
Mechanism and worked example
A checkout request has a 500 ms deadline. Allocate 80 ms local, 150 ms inventory, 180 ms payment and 90 ms response/network reserve. Do not give each dependency a fresh 500 ms timeout. If payment is unhealthy, fail safely or return pending rather than flooding it.
| Pattern | Protects against | Guardrail |
|---|---|---|
| Circuit breaker | Repeated calls to unhealthy dependency | Half-open probe budget |
| Bulkhead | One workload exhausting all resources | Per-pool concurrency and queue bounds |
| Retry with jitter | Transient failures | Attempt cap, deadline and idempotency |
| Timeout | Unbounded waiting | Propagate remaining budget |
Failure modes and operations
Three layers each making three attempts can multiply one request into 27 downstream attempts. Half-open breakers can stampede if every instance probes together. A timeout retry can duplicate a committed mutation. Bulkheads with unbounded queues hide overload.
Trade-offs and design boundary
Breakers fail fast but can reject during recovery. Bulkheads reduce blast radius but reserve capacity. Retries improve transient success while consuming capacity and tail latency. Assign one retry owner where possible.
Interview transfer exercise
Payment authorization times out. Explain how to retry. Reuse provider idempotency identity and query status after unknown outcome; never mint a new payment identity just because the response was lost.
Read the diagram
Caller propagates end-to-end deadline → Bulkhead admits bounded concurrency → Circuit breaker allows or rejects call → Retry backs off only safe transient failures