SLOs, Observability and Capacity

SLOs, Observability and Capacity

An architecture is easier to defend when its promises can be measured from the user's operation. Observability supplies evidence about state and behavior; an SLO defines which outcome matters and how much failure the product accepts. CPU graphs and successful queue acknowledgements are useful diagnostics, but neither alone establishes that a customer's order or notification completed correctly.

Define the operation before its objective

This original exercise has a notification service with two separate journeys: accepting a notification intent and submitting each eligible delivery to a provider. A 202 response means durable acceptance under the API contract, not that the recipient received a message. The second journey needs its own terminal-state accounting. Provider acceptance, device delivery and human reading are different milestones.

Fix the population and exclusions before calculating the result. A team cannot improve the SLO by relabeling overloaded valid requests as bad clients. Rejected abuse and invalid authentication still need separate security/abuse signals. If a client times out after a commit, the durable intent can exist while the request experience is bad. Deduplicated business effects and successful transport attempts answer different questions.

For this exercise, use a 99.9% request-based acceptance objective over a rolling 30 days. Specify retry counting: the acceptance SLI measures eligible HTTP attempts seen at the boundary, while unique-intent and per-delivery outcome metrics track business work. Do not mix their denominators. A later successful retry does not erase an earlier user-visible timeout from the attempt SLI.

Calculate the error budget with units

One million eligible attempts at 99.9% permits 1,000 bad attempts. If 250 have already occurred, 750 remain for the measured window at that volume. The final window's permitted count changes with its actual eligible volume; this is a retrospective count, not an entitlement to a fixed future number of errors.

A time-based 99.9% availability objective over 30 days permits 43.2 minutes of bad time under its own measurement definition. That is a different SLI. One minute during peak traffic can contain far more failed requests than one minute overnight. Do not substitute a time allowance for a request budget without changing the contract.

allowed_bad_fraction = 1 - objective
observed_bad_fraction = bad_eligible_events / all_eligible_events
burn_rate = observed_bad_fraction / allowed_bad_fraction
request_budget_remaining = eligible_count * allowed_bad_fraction - bad_count

With 200 bad attempts out of 10,000, the observed error fraction is 2%. Against a 0.1% allowance, the burn rate is 20. That means this window's error fraction is twenty times the permitted average, not that a 30-day budget will certainly expire at a predictable wall-clock time under changing traffic.

Alert on actionable sustained harm

An illustrative alert requires both short-window and long-window burn evidence above selected thresholds, sufficient eligible samples and a current data watermark. A brief old spike should not page after recovery solely because it remains inside a long window. A small service with two requests needs explicit low-volume behavior; a ratio based on one error can be meaningful, but noisy thresholds must be chosen deliberately.

Read the diagram

Count eligible attempts at the API boundary → Bad fraction is 2 percent; allowance is 0.1 percent → Fresh short and long windows establish impact → Operator bounds admission and dependency work → Recheck burn and accepted-work outcomes

View full sizeOpen original

SLOs, Observability and Capacity

Reset viewClose

Link the user-visible symptom to a reversible operational action and then verify recovery.

For a reproducible exercise, select a five-minute window requiring burn at least 14 with at least 100 eligible attempts, and a one-hour window requiring burn at least 6 with at least 1,000 attempts. Both measurements must be at most 120 seconds old. Short 200/10,000 yields 20x, while long 400/100,000 yields 4x: this pair does not page. If the long window reaches 800/100,000 (8x) while the short window still burns at 20x, it pages. A missing or stale input returns unknown, not a healthy zero. These custom numbers illustrate multi-window reasoning; they are not prescribed production defaults.

The repository example uses deliberately chosen thresholds for a deterministic exercise, not universal production paging defaults. A real alert needs an owner, a runbook, escalation and a clear condition for resolving the incident. Test stale data, missing collectors, zero traffic, counter resets and deployment transitions. “No samples” is not the same result as “zero bad events.”

Use metrics, logs and traces for different questions

Propagate trace context across supported HTTP/queue boundaries and correlate logs with it, but keep operation ID, occurrence ID, attempt ID and trace ID conceptually separate. One business operation can create several traces across retries and days. Trace baggage is not trusted authorization, and secrets or personal message content should not be propagated through it. Collector access and telemetry export are security boundaries too.

For a queue, trace enqueue-to-start and execution separately. A fast producer span can coexist with hours of waiting. Record due time, accepted time, start and terminal outcome under defined clock semantics. If timestamps originate from different machines, account for skew; a negative measured duration is an instrumentation problem to investigate, not a fast success.

Aggregate distributions, not instance percentiles

Two instances each serve 100 requests: all A requests take 10 ms; all B requests take 1,000 ms. Their p99 values are 10 and 1,000 ms. Averaging them gives 505 ms, while the combined nearest-rank p99 is 1,000 ms. Summing compatible histogram counts and estimating the combined quantile preserves a meaningful population; averaging quantile outputs generally does not. Record the bucket/schema/version compatibility and approximation error.

Counts and sums can support weighted averages. Two latency buckets with sums/counts 20/2 and 90/3 combine to mean 22, not the mean 20 of the two bucket averages. Histograms support other distribution questions; sampled traces alone are not an authoritative SLO denominator unless the sampling/estimation method explicitly supports it.

Capacity signals must change an operational decision

At 500 requests/s and 20 ms average slot occupancy, the stable average in-flight count is about 10. At 400 ms it becomes about 200. A configured 100-slot pool cannot sustain that second workload unchanged. Observe saturation, queue age and useful completions before adding frontend replicas that multiply dependency calls.

At 800 completions/s and 1,200 arrivals/s for a minute, backlog grows by 24,000. If arrivals then fall to 500/s, ideal drain takes 80 seconds at 300/s net. A deadline misses before disk fills if the waiting time exceeds the user's objective. Track predicted/observed deadline misses, not only queue length.

A metric with 20 route values, 3 regions and 4 outcome classes has up to 240 series before other dimensions. Adding one million user IDs can multiply that bound to 240 million. This arithmetic is a maximum Cartesian product, not a promise every combination exists. Use stable route templates and bounded dimensions, with secured, bounded diagnostic queries for individual identities instead of unbounded public metric labels.

Work an incident end to end

The acceptance API remains fast, but the dispatch milestone exceeds 60 seconds for one provider. Check accepted delivery age and provider-specific quota/latency, not only API success. Sample traces identify slow submissions; durable records distinguish unknown provider results from unsent work. Bound new dispatch, protect interactive status reads and reconcile original delivery identities. Observe whether both short and long error windows recover, whether oldest work drains and whether privacy/consent checks still hold under fallback.

If the collector itself is down, expose telemetry freshness and independent health probes. Avoid a monitoring backend that overloads the service with unrestricted emergency scans. Keep diagnostic retention and expensive query concurrency finite, especially during incidents.

Check your understanding

Question: A 202 returned in 100 ms, but a delivery remained unsubmitted for ten minutes. Was the dispatch objective met?

Question: A short error window is missing and the long window is stale. Should the display show zero burn?

Question: Instance p99 values are 10 and 1,000 ms with 100 requests at each constant latency. What is the combined nearest-rank p99?

Transfer exercise and scoring

For Metrics Monitoring, Centralized Logging and Notification System, define the user milestone, eligible denominator/exclusions, measurement source, alert response and a finite diagnostic resource budget. Score one point for each justified decision. Include a missing-telemetry case before calling the alert complete.

Prerequisites: Numbers to Know and Reliability and Overload Control.

Primary references and scope

• Google SRE Workbook: alerting on SLOs for error-budget-based alert design.
• OpenTelemetry context propagation for correlating signals and handling sensitive context.

The service, thresholds, populations, workloads and incident above are original exercises. Model arithmetic and alert-state tests do not establish production SLO compliance or replace a load test.