Learning pathsPricingChecking sign-in…
← ML curriculum

AIF-009 · FOUNDATION

AI runtime observability and SLOs

AI runtime observability connects request, retrieval, model, stream, tool and settlement stages into one privacy-aware trace. Useful operations require stage-level latency and token/cost evidence, quality and safety signals, explicit SLOs, and runbooks that distinguish provider faults from application defects.

Author self-review; independent technical review pending. Deterministic conceptual fixture; no live provider calls, production credentials or billing.

Prerequisites and scoped previews

Hybrid retrieval, reranking and grounded evidence

Combine dense and lexical retrieval, fuse candidates, enforce permissions, rerank bounded evidence and evaluate grounded answers.

Streaming, cancellation and backpressure

Design resumable ordered generation streams, cancellation fencing, bounded buffers and usage settlement.

Model routing, fallbacks and unit economics

Route only to eligible models, distinguish retry from fallback and optimize cost per successful task without violating quality or policy.

What, when, why and how

Trace the logical run across asynchronous boundaries

what
A trace correlates admission, retrieval, prompt assembly, provider attempts, stream events, tools and usage settlement under one logical run.
when
Use it for multi-stage RAG, agent workflows, streaming and provider routing.
why
A single HTTP duration hides queue wait, retrieval, time-to-first-token, generation and post-processing delays.
how
Propagate trace/run IDs and bounded span attributes across queues and workers; link fallback attempts without losing the parent run.

Measure latency as a budget

what
End-to-end latency decomposes into admission, queue, retrieval, prompt assembly, provider setup, time-to-first-token, inter-token gaps and finalization.
when
Use it to set user-facing SLOs and identify bottlenecks by workload and provider.
why
Optimizing average model latency can worsen user experience if queueing or retrieval dominates p95.
how
Record histograms and percentiles by model, route, tenant tier and task; compare each stage with its allocated deadline budget.

Token and cost telemetry needs semantic meaning

what
Input, output, cached, reasoning and tool tokens are distinct usage dimensions with a versioned price schedule.
when
Use it for tenant budgets, cost attribution, unit economics and anomaly detection.
why
Character counts and estimated tokens are not authoritative billing evidence; late provider usage can differ from streamed text.
how
Store estimates separately from terminal provider-reported usage, price with the pinned schedule and reconcile once.

1. Follow one logical request end to end

Create a trace at authenticated admission and attach a stable run ID. Child spans cover retrieval, authorization, context construction, provider attempt, first token, stream delivery, tool execution and usage settlement. Queue boundaries propagate trace context explicitly; fallback attempts are linked as children or related runs rather than replacing the original history.

Use low-cardinality dimensions for metrics—model family, route, outcome, region and task class. Put high-cardinality run IDs in traces and logs, not metric labels. Never put raw prompts, retrieved documents, credentials or personal data into default telemetry.

VISUAL WALKTHROUGH

See the system in motion

Follow a logical AI request across admission, retrieval, model execution, streaming and settlement.

Process overviewAvailable even when interactive playback cannot load.
  1. Create trace and logical run ID

  2. Measure queue and retrieval stages

  3. Record provider attempts and TTFT

  4. Correlate stream and tool outcomes

  5. Reconcile authoritative usage

  6. Evaluate quality and SLO guardrails

Loading interactive walkthrough…

2. Decompose latency and define SLOs

Measure time to admission, queue wait, retrieval, context build, time-to-first-token (TTFT), inter-token latency, total generation and finalization separately. Track p50/p95/p99 and timeout/cancellation rates by workload. TTFT and total completion latency are different user experiences and should not be collapsed into one average.

Set an end-to-end deadline and allocate stage budgets. A retry consumes the remaining budget. If retrieval has exhausted the deadline, do not start an expensive generation that cannot finish within the product SLO.

3. Reconcile tokens and cost from authoritative evidence

Record estimated token counts for admission and provider-reported terminal usage separately. Attribute input, output, cached and reasoning tokens where available; include tool, embedding and reranking costs in the task ledger. Pin the price schedule version to the run so historical cost can be reproduced after provider pricing changes.

Streaming deltas are not a billing ledger. Providers can report usage after the final text chunk, and cancellation may produce partial usage. Reconcile late and duplicate usage reports idempotently, and alert on large estimate-to-actual variance.

4. Connect reliability to quality and safety

A fast response can still be wrong, ungrounded or unsafe. Pair runtime metrics with retrieval recall, citation support, structured-output validity, tool success, refusal correctness and user correction signals. Slice by language, task, model, route and customer tier; avoid a single aggregate that hides regressions.

Define runbooks for provider outage, elevated 429s, TTFT regression, cost spikes, empty retrieval, invalid tool arguments and delayed usage settlement. Each alert needs an owner, a safe mitigation and a rollback or disable path.

Diagrams and relationships

AI runtime observability trace

One logical run correlates stage latency, provider attempts, streamed output, quality signals and authoritative usage settlement.

Scroll horizontally on smaller screens, or enlarge the diagram.

AI runtime observability traceOne logical run correlates stage latency, provider attempts, streamed output, quality signals and authoritative usage settlement. Admission + trace to Retrieve + context: trace context. Retrieve + context to Provider attempt: stage spans. Provider attempt to Stream + tools: TTFT + deltas. Stream + tools to Quality + safety: outcome signals. Provider attempt to Usage + cost ledger: terminal usage.trace contextstage spansTTFT + deltasoutcome signalsterminal usageAdmission + traceRetrieve + contextProvider attemptStream + toolsQuality + safetyUsage + cost ledger
AI runtime observability traceOne logical run correlates stage latency, provider attempts, streamed output, quality signals and authoritative usage settlement. Admission + trace to Retrieve + context: trace context. Retrieve + context to Provider attempt: stage spans. Provider attempt to Stream + tools: TTFT + deltas. Stream + tools to Quality + safety: outcome signals. Provider attempt to Usage + cost ledger: terminal usage.trace contextstage spansTTFT + deltasoutcome signalsterminal usageAdmission + traceRetrieve + contextProvider attemptStream + toolsQuality + safetyUsage + cost ledger

Engineer, Senior and Staff follow-ups

These are teaching lenses. Each answer changes the scope of reasoning; they are not official employer leveling criteria.

Engineer: Why should run IDs not be metric labels?

Run IDs have high cardinality and can explode metric storage. Keep them in traces/logs and use bounded dimensions such as model family, region, task class and outcome for metrics.

Engineer: What is the difference between TTFT and total latency?

TTFT measures how long until the user sees the first generated token; total latency measures completion. They capture different experience and capacity behaviors.

Senior: How do you reconcile token usage after a cancelled stream?

Store estimates separately, wait for terminal provider/worker usage evidence, deduplicate reports by attempt and usage identity, apply the pinned price schedule and settle once.

Senior: What should an AI runtime dashboard show?

Request success and timeout rates, queue wait, retrieval duration, TTFT, inter-token latency, total latency, token dimensions, cost per successful task, provider errors, tool outcomes and quality/safety guardrails.

Staff: How do you design privacy-safe observability?

Use trace IDs and minimized structured attributes, redact secrets and personal data, keep raw content off by default, enforce access/retention controls and sample traces under explicit policy.

Staff: How do you connect SLOs to release decisions?

Use stage-level reliability and latency SLOs alongside quality/safety floors and cost budgets. Canary by workload slice, block promotion on statistically meaningful regressions and preserve a fast rollback/disable path.

Practice and self-review

Your practice draft

An AI assistant's p95 latency doubled, cost per successful task rose 30%, but provider latency dashboards look unchanged. Design the investigation and telemetry needed to find the cause.

Saved locally on this device. The worked answer does not score your draft. Existing lesson notes and completion are separate.

Progressive hints

Hint 1

Decompose latency across queue, retrieval, context, TTFT, generation and finalization.

Hint 2

Separate estimated from terminal token usage and include retries/tools/retrieval costs.

Hint 3

Correlate quality, route, provider attempt and tenant slices under one logical run.

Reveal worked answer guidance

Start from logical-run traces and compare stage histograms against the previous release. Check queue wait, retrieval fan-out, context token count, cache hit rate, TTFT, inter-token latency, retries, fallback frequency and tool calls. Compare estimated versus terminal input/output/cached/reasoning usage using the pinned price schedule. Slice by model, route, task, tenant tier and release. Verify answer quality and tool success so a cheaper/faster but degraded route is not misclassified as an optimization. Mitigate the identified stage with bounded concurrency, route rollback or feature disable, then add an SLO and regression gate.

Evidence to check in your answer

  • End-to-end latency is decomposed into stages and percentiles.
  • Cost is based on reconciled usage and all relevant dependencies.
  • Signals are correlated by run and sliced without high-cardinality metrics.
  • Quality/safety guardrails and mitigation are included.

For each item, record Missing, Prompted, Independent or Adaptive evidence. Adaptive means you can revise the reasoning under a changed constraint. Reading completion and readiness are separate.

Sources and verification scope

Original synthetic examples and reference calculations. Sources support narrowly scoped background; they do not establish production performance or independent chapter review.

SRE workbook: data processing pipelines ↗ · Pipeline SLOs and operational ownership; do not copy source workload numbers as course production results.

Hidden Technical Debt in Machine Learning Systems ↗ · System-boundary and lifecycle risks; source lessons motivate original examples.

vLLM automatic prefix caching ↗ · Prefix KV reuse and its prefill-specific benefit; does not establish tenant authorization or cross-tenant cache policy.

OpenTelemetry GenAI token metrics ↗ · Token usage counters versus per-operation histograms; use billable usage when both billable and consumed usage are available.

Primary background links checked during authoring on 2026-10-19. Arithmetic, temporal eligibility, maturity and split reference functions are checked by the bounded local fixture suite. SQL is explanatory, not a migrated production query. No GPU, external dataset, model API or training job is required.