Tail Latency and Hedged Requests

Tail Latency and Hedged Requests

Tail latency dominates user experience when a request waits on many dependencies or fans out to many workers. Measure p50, p95 and p99 per operation. A request waiting for all n independent branches completes at the maximum branch latency; as n grows, slow-tail exposure rises. Hedged requests send a duplicate to another replica after a delay and accept the first valid result, but must cancel or safely ignore the loser.

Mechanism and worked example

If a page needs 20 independent shard responses, even a small per-shard slow-tail probability can raise page p99. Parallelizing reduces sum latency but does not remove maximum-tail behavior. Bound fan-out, allow explicit partial results and share one deadline.

Failure modes and operations

Hedging can double load during overload. Coordinated omission hides queueing if measurements record only completed requests. Retries amplify tail latency. Slow dependencies can hold threads and connections after the caller deadline.

Trade-offs and design boundary

Hedging can reduce latency when spare capacity exists and operations are idempotent/read-only. It is harmful when replicas share a bottleneck or requests are expensive. Partial responses help only when missing data is acceptable and represented honestly.

Interview transfer exercise

Design a search aggregator with bounded fan-out, one deadline, explicit partial completeness and a small hedge budget only when spare capacity is measured.

Read the diagram

Request fans out to bounded shards → Each branch shares one deadline → Slow branch may receive delayed hedge → Aggregator returns complete or explicit partial result

View full sizeOpen original

Tail Latency and Hedged Requests

Reset viewClose

Mechanism and decision flow for Tail Latency and Hedged Requests.