Reliability and Overload Control
Bound deadlines, queues, concurrency and retry work; calculate safe admission and recovery.
Reliability and Overload Control
A service is reliable only within a defined workload and failure policy. Queues, retries and more workers can help short disruptions while making sustained overload worse. Begin with an end-to-end objective and finite budgets for time, concurrency, memory, storage and retries. Decide which work to reject or degrade before resources are exhausted.
Trace one request budget
This original exercise has a one-second response deadline for an interactive request. Illustrative allocations are 100 ms for admission/queueing, 150 ms for application work, up to 600 ms for a dependency and 150 ms for response handling and margin. These are planning caps, not universal latency constants. Actual elapsed time, not the original allocation, determines the remaining deadline at each step.
| Situation | Decision |
|---|---|
| Request waited 250 ms | Only 750 ms remains; do not grant the dependency a new full second |
| Remaining budget is less than useful service time | Reject or degrade rather than start predictably stale work |
| Caller disconnected | Propagate cancellation where supported; irreversible effects may already exist |
| Dependency timed out after a mutation | Outcome is unknown; resolve using the original operation identity |
Set connection and request timeouts and understand what each covers. A local monotonic clock measures elapsed time without wall-clock adjustments. Across hosts, propagate the deadline using a supported protocol with understood clock assumptions, or conservatively propagate remaining budgets; never assume all machines have identical clocks.
Concurrency is a separate limit
Suppose arrivals are 500/s and mean time occupying a dependency slot is 20 ms. In stable conditions the average occupancy is about 10. If that time rises to 400 ms at the same arrival rate, the implied average is 200. A pool of 100 cannot sustain this workload indefinitely. Little's Law describes average stable behavior; it is not permission to size exclusively from means or ignore tail latency.
Use a bounded admission queue and a dependency semaphore. Separate interactive work from bulk exports and divide budgets by tenant or class. This is a bulkhead: one slow dependency or large customer cannot consume every worker. A global semaphore inside each process is only process-local; across 20 replicas, 100 slots per replica permits up to 2,000 calls. The downstream contract needs an aggregate budget or safe allocation.
Calculate queue stability before selecting a broker
Assume 1,200 admitted jobs/s, a service ceiling of 800/s and a 60-second burst. The backlog grows by 24,000 jobs. At 2 KB/job it represents 48 MB of payload before broker overhead. The tail job has roughly 30 seconds of waiting at an 800/s service rate even if the queue fits in memory. A two-second job-start objective therefore fails long before storage fills.
When arrivals fall to 500/s and service remains 800/s, net drain is 300/s, so ideal catch-up takes 80 seconds. If arrivals remain 1,200/s, recovery capacity of 800/s can never drain the queue. Additional workers help only when the downstream system safely supports more useful throughput.
Read the diagram
Rejection protects the finite system; recovery requires completion capacity greater than admitted arrivals.
A queue is useful for accepted durable work with an explicit completion objective. It is a poor place to hide interactive requests whose deadlines will expire before service. Measure oldest-job age and remaining deadline as well as depth and bytes.
Make retries consume a budget
If each of five layers independently permits three attempts, the leaf can receive up to 243 attempts for one original request in a worst-case retry tree. That is not a prediction of normal traffic, but it demonstrates amplification. Prefer a clearly owned retry policy; account for retries in the same downstream capacity budget as initial attempts.
For a full-jitter policy, attempt n can wait a random duration in [0, min(cap, base * 2^n)]. Bound the exponent, total attempts and elapsed deadline. Sample with an injected random source in tests. Jitter spreads retries; it does not make an overloaded dependency faster or create missing capacity.
A local retry budget allowing 100 initial requests and 20 extra attempts can cap this interval at 120 attempts. Define how tokens replenish and how budgets combine across instances; do not pretend this simple interval model is a production rate limiter. Respect a supported Retry-After instruction within the caller's deadline and retry only eligible operations with stable intent identity.
Distinguish protective mechanisms
| Mechanism | Protects | Does not guarantee |
|---|---|---|
| Rate limit | Arrival budget per chosen identity/time policy | Bounded concurrent occupancy when service slows |
| Concurrency limit | Active dependency work | Fairness without scheduling or admission rules |
| Bounded queue | Waiting bytes/items | Meeting deadlines or absorbing indefinite overload |
| Circuit breaker | Avoiding repeated calls to a failing dependency | More capacity, correct mutation retries or safe recovery by itself |
| Load shedding | Preserving useful admitted work | Success for the rejected request |
| Backpressure | Communicating reduced capacity upstream | That every upstream producer will obey |
A breaker can move from closed to open, then admit a bounded number of half-open probes after a recovery interval. Without probe limits, every process can test recovery at once and recreate the outage. Keep its state scope explicit. Trip criteria should distinguish dependency failure from ordinary business rejection and should consider sample volume. The chosen library's exact semantics matter.
Choose a degradation policy that preserves the invariant
A public recommendation page may fall back to permitted popular items. A balance mutation cannot invent a successful debit when its authority is unavailable. Cached content still needs the privacy and freshness policy its product promises. Optional analytics can sometimes be sampled or deferred, while accepted financial work needs durable accounting and a clear refusal policy before acceptance.
Reject at the earliest useful boundary, return an honest capacity error and avoid expensive work before admission. For already accepted durable jobs, apply the documented deadline, retry and repair policy; do not silently discard them to make the queue graph look healthy. Tenant isolation and weighted scheduling can preserve useful service, but must be sized against the actual admitted workload.
Recovery is another traffic event
When a dependency returns, fresh requests, delayed jobs and retries compete. Reserve room for recovery and use gradual admission, bounded replay and jittered reconnects. Reducing a queue from 24,000 to zero by flooding the recovering database can cause another outage. Also inspect autoscaling delay and downstream connection pools: fast API scaling can multiply database pressure.
Track admitted/rejected requests, successful useful completions, queue age, deadline misses, active slots, retries per original operation and dependency latency percentiles. Report rejection as a customer outcome, not as hidden success. Alert on the agreed objective, not just CPU usage.
Check your understanding
Question: An 800/s consumer receives 1,200/s for one minute. How large is the backlog, and will an infinite queue fix the throughput deficit?
Show answer and explanation
Answer: Backlog grows by 24,000 jobs. No queue size increases completion throughput. A finite admission/retention policy and enough recovery capacity are required.
Question: A 100-slot process is replicated 20 times. Does the dependency still see at most 100 concurrent calls?
Show answer and explanation
Answer: No. Independent local limits allow up to 2,000. Allocate an aggregate safe budget or coordinate it using a mechanism whose own failure behavior is defined.
Question: A mutation timed out while the circuit opened. Is submitting a fresh operation when the breaker closes safe?
Show answer and explanation
Answer: The first operation may have committed. Reuse its operation identity or resolve status; breaker recovery does not resolve business uncertainty.
Transfer exercise and scoring
For Priority Job Queue, give arrival/service assumptions, a burst duration, backlog/drain arithmetic, rejection policy and a tenant fairness rule. Score one point per justified item. Then apply the same budget to Notification System and API Gateway, accounting for provider quotas and all replicas.
Prerequisites: Numbers to Know, Scaling Writes and Idempotency and Delivery Semantics.
Primary references and scope
- AWS Builders' Library: timeouts, retries and jitter.
- AWS Architecture Blog: exponential backoff and jitter.
The deadlines, workload, interval budget, queue model and scored exercises are original assumptions. Tests check finite-resource behavior and arithmetic, not real network timing or production throughput.