Prerequisites and scoped previews
Labels, delayed outcomes and human annotation
Define label windows, maturity, censoring, annotation policy and disagreement.
Leakage, splits and evaluation contamination
Prevent future information, entity overlap, preprocessing leakage and test-set reuse.
Classification and rare-event evaluation
Define confusion counts, precision/recall, PR/ROC metrics, slices and threshold trade-offs.
Retrieval and ranking evaluation
Define recall/precision at K, MRR, DCG/NDCG, relevance labels and query/user weighting.
Slices, fairness and harm-aware decisions
Define relevant slices and competing error costs without claiming one metric certifies fairness.
AI coding assistant with repository context and safe tool execution
Commit-pinned code retrieval, safe patch generation, isolated validation and approval-gated repository writes.
Multi-tenant LLM inference gateway
Tenant-safe model routing, atomic budget admission, KV-aware serving, resumable streams and evidence-based usage settlement.
What, when, why and how
Evaluation is a system, not a single score
- what
- AI quality spans task success, correctness, groundedness, retrieval, safety, latency, cost and user outcomes.
- when
- Before model, prompt, retrieval or agent changes and continuously in production.
- why
- A single aggregate score hides regressions and rewards fluent but incorrect output.
- how
- Define decision-specific metrics, slices, thresholds and evidence provenance.
Version the entire evaluation unit
- what
- A result depends on dataset, labels, prompt, model, tools, retrieval index, rubric and evaluator versions.
- when
- For reproducible comparisons, audits and rollback.
- why
- Changing any dependency changes the output or its interpretation.
- how
- Create immutable evaluation manifests and content-addressed artifacts.
Separate offline evaluation from online outcomes
- what
- Offline replay measures controlled quality; online telemetry and experiments measure behavior under real traffic.
- when
- Before release and after deployment.
- why
- Offline datasets cannot fully represent live users or distribution shift.
- how
- Use offline gates, canaries, guarded experiments and privacy-safe production feedback.
Use metric families, not metric substitution
- what
- Retrieval recall, answer correctness, groundedness, citation validity, abstention and task success are distinct.
- when
- For RAG, generation and agentic workflows.
- why
- A strong retrieval score does not prove a correct answer.
- how
- Score each stage and end-to-end outcome separately; inspect failure slices.
Treat model judges as fallible instruments
- what
- LLM judges have bias, variance, position effects, rubric sensitivity and correlated errors.
- when
- At scale when human review alone is too expensive.
- why
- Uncalibrated judges create false confidence and silently move release gates.
- how
- Calibrate against blinded human labels, measure agreement and uncertainty, route ambiguity to humans.
Close the feedback loop with governance
- what
- User feedback, support tickets and production traces become candidate examples only after privacy, provenance, deduplication and review.
- when
- For continuous quality improvement.
- why
- Raw feedback is noisy, biased, sensitive and sometimes adversarial.
- how
- Quarantine, redact, label, adjudicate, version and evaluate before promotion.
Make release decisions evidence-based
- what
- A release gate combines metric thresholds, confidence intervals, slice guardrails, safety vetoes and operational SLOs.
- when
- For prompt/model/index/tool changes.
- why
- Aggregate averages hide severe regressions for a tenant, locale or high-risk task.
- how
- Compare candidates on paired examples, enforce sample sufficiency and retain rollback artifacts.
Frame the evaluation and feedback platform
Design a platform that helps product and AI teams decide whether a model, prompt, retrieval configuration, tool policy or application release is better, safe enough and operationally viable. It supports repeatable offline experiments, online monitoring, human review, model-judge assistance, regression gates and governed feedback ingestion.
Synthetic exercise assumptions: 12 teams, 40 AI applications, 30 candidate evaluations/day, 25,000 examples in the shared corpus and 2 million production traces/day before sampling. These are interview inputs, not measured traffic.
Define requirements, trust boundaries and SLOs
Functional requirements: register immutable datasets and rubrics; create evaluation runs; replay pinned candidates; compute stage and end-to-end metrics; compare baseline/candidate; run calibrated judges; collect human annotations; enforce release gates; ingest reviewed feedback; export audit evidence.
Track run completion latency, queue delay, cost per example, evaluator agreement, human-review turnaround, gate error rates and production quality drift. Keep raw prompts and outputs out of general analytics by default.
Define the evaluation contract and unit of comparison
An evaluation case has stable case ID, input, expected evidence/outcome, task type, tenant-safe metadata, rubric version and adversarial tags. Candidate manifest pins application build, prompt, model/provider, decoding, tools/policies, retrieval corpus/index, embedding and reranker versions.
Store structured observations, metric values, evidence references, evaluator identity/version, timestamps, confidence and execution status. Immutable run IDs and hashes make results explainable.
Version datasets, labels and slices
Validate schema, provenance, licensing/consent, tenant scope and duplicate fingerprints. Dataset corrections create new immutable versions; they never rewrite historical results. Keep development, holdout and red-team partitions separated.
Represent slices such as language, task type, difficulty, document freshness, safety category and tool path. Require minimum sample counts and protect small sensitive cohorts.
VISUAL WALKTHROUGH
See the system in motion
Version governed evaluation data without mutating historical evidence.
Validate source and consent
Normalize, redact and deduplicate
Publish immutable dataset version
Pin dataset and rubric in run manifest
Loading interactive walkthrough…
Replay candidate systems deterministically where possible
Resolve a pinned candidate manifest and create a durable run with bounded shards. Execute each case through a sandboxed adapter with strict timeouts, concurrency limits, budgets and idempotency keys. Capture provider, retrieval and tool evidence without secrets.
Hosted model output may not be bit-identical. Preserve inputs, exposed model version, sampling settings, retrieval snapshot, fixtures and raw output under retention policy. Claim traceability, not guaranteed determinism.
VISUAL WALKTHROUGH
See the system in motion
Replay a pinned candidate through durable bounded shards.
Resolve immutable candidate and dataset
Dispatch bounded idempotent shards
Replay cases with deadlines
Persist provenance-rich results
Loading interactive walkthrough…
Measure retrieval independently
For RAG, measure recall@K, precision@K, reciprocal rank or nDCG where relevance judgments exist, plus ACL violation rate and freshness. Separate candidate generation from reranking.
A high recall score does not establish answer correctness. An unauthorized candidate is a release-blocking security failure even if ranking metrics look good.
VISUAL WALKTHROUGH
See the system in motion
Measure retrieval, generation, safety and task outcomes independently.
Measure retrieval relevance and ACL correctness
Score answer correctness and grounding
Run safety and policy checks
Report metric slices and uncertainty
Loading interactive walkthrough…
Measure answer quality, grounding and abstention
Score factual correctness against trusted references, claim-level groundedness, citation precision/coverage, completeness, instruction adherence and appropriate abstention. Allow multiple acceptable answers and evidence sets.
Keep semantic similarity diagnostic, not a correctness substitute. For agents, add task completion, tool selection/argument correctness, policy compliance, side-effect correctness and recovery.
Evaluate safety and policy as independent gates
Build suites for prompt injection, data exfiltration, cross-tenant access, unsafe tool calls, PII disclosure, harmful output and refusal behavior. Include positive and negative cases so refusal-everything is not rewarded.
Safety thresholds are often non-compensatory: a serious authorization or side-effect violation can veto release despite better average quality.
Use model judges only with calibrated rubrics
A judge receives a versioned rubric, bounded evidence and structured scoring schema. Prefer pairwise blinded comparisons for preference tasks; randomize candidate order and test position bias. Pin judge model, prompt and decoding.
Track agreement with adjudicated human labels, confusion by slice, calibration, position sensitivity and disagreement. Do not let the candidate model family be the sole judge of itself.
VISUAL WALKTHROUGH
See the system in motion
Calibrate model judges against blinded human labels.
Blind and randomize candidate order
Apply versioned judge rubric
Compare with adjudicated labels
Route uncertain cases
Loading interactive walkthrough…
Route uncertain and high-risk cases to human review
Use stratified sampling, disagreement queues, low-confidence thresholds and severity escalation. Mask sensitive fields by reviewer role; blind candidate identity where practical.
Support double annotation and adjudication for critical labels. Measure inter-annotator agreement without treating agreement as truth; retain reviewer training and rubric history.
Turn production feedback into governed examples
Ingest explicit ratings, corrected answers, support tickets, escalations, task failures and sampled traces through quarantine. Verify source, consent and tenant scope; redact, deduplicate, attach provenance and enforce retention.
Never auto-promote raw feedback into a gold dataset. Triage, label, adjudicate and review for poisoning, manipulation and selection bias.
VISUAL WALKTHROUGH
See the system in motion
Quarantine and review production feedback before dataset promotion.
Capture minimal feedback
Quarantine and verify provenance
Redact, deduplicate and detect poisoning
Review and adjudicate
Loading interactive walkthrough…
Build durable asynchronous evaluation runs
Persist run state and shard checkpoints, dispatch bounded work through queues, lease shards and make case execution idempotent. Separate orchestration metadata from large artifacts in object storage.
Use queued, running, partial, completed, failed, cancelled and expired states. Retry transient provider failures with stage budgets; partial runs are never release evidence.
Compare candidates with uncertainty and paired analysis
Prefer paired evaluation: run baseline and candidate on the same cases and compare per-case deltas. Report sample size, mean/median, confidence intervals and slice-level changes.
Predefine primary metrics, guardrails, sample sizes and stopping rules. Avoid tiny samples and repeated peeking; account for multiple comparisons across many slices.
Gate releases and preserve rollback evidence
Evaluate absolute floors, candidate deltas, confidence bounds, critical slices, safety vetoes and operational constraints. A gate can pass, fail or require human review; unknown or underpowered evidence is not a pass.
Bind approval to exact candidate manifest, dataset versions, run IDs and gate policy. Promote through canary and retain the previous known-good configuration.
VISUAL WALKTHROUGH
See the system in motion
Gate promotion on paired evidence, sample sufficiency and safety.
Compare paired cases
Check sample sufficiency
Apply safety and slice floors
Canary with rollback
Loading interactive walkthrough…
Monitor live quality without collecting everything
Combine privacy-safe aggregates, sampled traces, explicit feedback and operational telemetry. Track task mix, retrieval coverage, refusal rates, judge/human disagreement, tool failures, latency and cost.
Feedback is self-selected and sparse. Use representative sampling and periodic human audits; distinguish satisfaction, task outcome and technical quality.
Handle failures, stale runs and evaluator drift
Provider outages, rate limits, worker crashes, dataset corruption, judge changes and annotation backlogs are distinct failure modes. Persist checkpoints and retry only transient failures.
Fence workers with run/shard generations; stale workers cannot overwrite newer results. A judge or rubric change needs a calibration bridge before scores are comparable.
Enforce privacy, tenancy and auditability
Authorize every dataset, run, artifact, annotation and export by tenant and role. Use scoped artifact URLs, encryption, audit events and explicit retention/deletion. Redact before external judging where policy requires.
Treat prompts, retrieved documents, feedback and outputs as untrusted. Separate rubric instructions from evidence and constrain evaluation tools to read-only fixtures.
Estimate capacity and control evaluation spend
Capacity is examples × candidates × repetitions × judge passes × tokens, plus human review. Apply quotas, priority classes, per-run budgets, token caps, concurrency pools and cancellation.
Cache immutable preprocessing only when keys include every relevant version and tenant boundary. Attribute cost by team, application, candidate and stage; reserve then reconcile usage.
Assemble the complete evaluation feedback platform
The control plane manages datasets, candidate manifests, rubrics, policies and approvals. The data plane executes durable runs, collects observations, computes metrics, calibrates judges and routes selected cases to humans. A release controller consumes evidence.
The production feedback plane samples privacy-safe events into quarantine, then routes reviewed examples into versioned datasets. Failure paths include provider outage, stale-worker fencing, judge drift and blocked releases.
Complete animated architecture walkthrough
Follow immutable evaluation inputs through durable replay, metric decomposition, judge calibration, human review, release gating and governed production feedback.
VISUAL WALKTHROUGH
See the system in motion
Trace the full evaluation lifecycle, including a blocked release and the feedback path into a reviewed dataset version.
Register immutable datasets, rubrics and candidate manifest
Schedule durable bounded replay jobs
Compute retrieval, generation, safety and task metrics
Calibrate judges and route uncertain cases to humans
Compare baseline and candidate with confidence and slice guardrails
Block unsafe or underpowered releases; approve exact evidence
Canary candidate and preserve rollback configuration
Quarantine production feedback, review it and publish a new dataset version
Loading interactive walkthrough…
Diagrams and relationships
Real-time media flows separately from reliable session and turn control.
Scroll horizontally on smaller screens, or enlarge the diagram.
Every downstream result is tied to a turn generation; interruption invalidates older output.
Scroll horizontally on smaller screens, or enlarge the diagram.
Speech intent is committed, authorized and confirmed before side effects.
Scroll horizontally on smaller screens, or enlarge the diagram.
Engineer, Senior and Staff follow-ups
These are teaching lenses. Each answer changes the scope of reasoning; they are not official employer leveling criteria.
Engineer: Why separate partial and final ASR events?
Partial hypotheses can be revised. Irreversible actions must use committed intent; speculative retrieval or drafting can be tied to a revision and discarded if it changes.
Engineer: Why use WebRTC instead of WebSocket?
WebRTC provides media-oriented transport features such as jitter handling and congestion adaptation. WebSocket can work for controlled frame streaming, but the application must build pacing, loss and backpressure behavior itself.
Senior: How do you make barge-in reliable?
Advance a turn generation, stop playback immediately, cancel model and synthesis work, and reject every late chunk whose generation is stale. Keep playback buffers bounded and account for acoustic echo.
Senior: How would you reduce time-to-first-audio?
Measure each stage, overlap safe low-risk work, use streaming ASR/LLM/TTS, tune endpointing, keep providers warm and bound buffers. Do not trade away tool safety or speak unstable transcript content.
Senior: How do you handle uncertain recognition of an amount or account?
Treat ASR confidence and semantic ambiguity as risk signals. Ask a clarification, read back critical values and require explicit confirmation before a high-impact action.
Staff: How would you support multiple ASR and TTS providers?
Define normalized streaming contracts, capability and language metadata, deadlines, cancellation semantics and usage reporting. Route per stage and preserve committed turn state across compatible fallback.
Staff: How do you prevent stale speech after interruption?
Use monotonic turn generations and include generation/sequence in every text and audio chunk. The playback gateway validates before enqueue and again before rendering.
Staff: How do you evaluate conversational quality?
Combine ASR slices, endpointing and interruption metrics, human listening, task completion, tool correctness, safety and user outcomes. Replay is useful but does not replace live network and interaction testing.
Practice and self-review
Your practice draft
Design a multi-tenant evaluation and feedback platform for RAG, LLM and agent applications. Support immutable datasets, candidate replay, metric computation, calibrated judges, human review, production feedback, regression gates and safe rollout.
Saved locally on this device. The worked answer does not score your draft. Existing lesson notes and completion are separate.
Progressive hints
Hint 1
Define the candidate manifest and immutable evaluation case first.
Hint 2
Separate retrieval, generation, safety, tool and product outcome metrics.
Hint 3
Make evaluation runs durable, sharded and idempotent.
Hint 4
Treat model judges as fallible and calibrate against blinded human review.
Hint 5
Quarantine and govern production feedback before dataset promotion.
Hint 6
Bind release decisions to exact evidence and preserve rollback artifacts.
Reveal worked answer guidance
Register immutable dataset and rubric versions, then create a candidate manifest pinning application, prompt, model, index, tools and policies. The orchestrator schedules bounded idempotent shards and records provenance-rich outputs. Metric workers separately score retrieval, grounded generation, citations, abstention, safety, tool correctness and task outcome. A calibrated judge handles scalable low-risk scoring while disagreement and high-risk cases route to blinded human review and adjudication. A comparison service performs paired baseline/candidate analysis with confidence intervals and slice guardrails. The release controller requires sample sufficiency, safety floors and operational budgets, binds approval to exact run evidence, then supports canary and rollback. Production traces enter privacy-aware quarantine, are redacted and reviewed before becoming versioned evaluation examples.
Evidence to check in your answer
- Dataset and candidate dependencies are immutable and fully versioned.
- Offline replay is durable, bounded, idempotent and provenance-rich.
- Retrieval, generation, safety, tool and product outcomes are measured separately.
- Judge reliability is calibrated against human labels and uncertainty is visible.
- Human review is blinded where practical and supports adjudication.
- Production feedback is quarantined, minimized, consent-aware and reviewed.
- Statistics include sample sufficiency, paired comparison and slice guardrails.
- Release gates bind exact evidence and include safety vetoes and rollback.
- Tenant isolation, privacy, retention and audit are explicit.
- Capacity, token cost, queueing and cancellation are addressed.
For each item, record Missing, Prompted, Independent or Adaptive evidence. Adaptive means you can revise the reasoning under a changed constraint. Reading completion and readiness are separate.
Sources and verification scope
Original synthetic examples and reference calculations. Sources support narrowly scoped background; they do not establish production performance or independent chapter review.
Primary background links checked during authoring on 2026-10-19. Arithmetic, temporal eligibility, maturity and split reference functions are checked by the bounded local fixture suite. SQL is explanatory, not a migrated production query. No GPU, external dataset, model API or training job is required.