AISD-006 · DESIGN PRACTICE
Practice: LLM evaluation and feedback platform
Draft the complete decision system before revealing the guidance. Use a 45-minute rehearsal budget and adapt it to the interviewer’s direction.
Preserve eligibility, available-at-decision inputs, mature labels, bounded work and compatible releases. Explain alternatives and test a normal, boundary and failure case.
Your practice draft
Design a multi-tenant evaluation and feedback platform for RAG, LLM and agent applications. Support immutable datasets, candidate replay, metric computation, calibrated judges, human review, production feedback, regression gates and safe rollout.
Saved locally on this device. The worked answer does not score your draft. Existing lesson notes and completion are separate.
Progressive hints
Hint 1
Define the candidate manifest and immutable evaluation case first.
Hint 2
Separate retrieval, generation, safety, tool and product outcome metrics.
Hint 3
Make evaluation runs durable, sharded and idempotent.
Hint 4
Treat model judges as fallible and calibrate against blinded human review.
Hint 5
Quarantine and govern production feedback before dataset promotion.
Hint 6
Bind release decisions to exact evidence and preserve rollback artifacts.
Self-review evidence
For each criterion, record Missing, Prompted, Independent or Adaptive reasoning. No automated scoring or offer prediction is provided.
- Dataset and candidate dependencies are immutable and fully versioned.
- Offline replay is durable, bounded, idempotent and provenance-rich.
- Retrieval, generation, safety, tool and product outcomes are measured separately.
- Judge reliability is calibrated against human labels and uncertainty is visible.
- Human review is blinded where practical and supports adjudication.
- Production feedback is quarantined, minimized, consent-aware and reviewed.
- Statistics include sample sufficiency, paired comparison and slice guardrails.
- Release gates bind exact evidence and include safety vetoes and rollback.
- Tenant isolation, privacy, retention and audit are explicit.
- Capacity, token cost, queueing and cancellation are addressed.
Reveal worked design guidance
Register immutable dataset and rubric versions, then create a candidate manifest pinning application, prompt, model, index, tools and policies. The orchestrator schedules bounded idempotent shards and records provenance-rich outputs. Metric workers separately score retrieval, grounded generation, citations, abstention, safety, tool correctness and task outcome. A calibrated judge handles scalable low-risk scoring while disagreement and high-risk cases route to blinded human review and adjudication. A comparison service performs paired baseline/candidate analysis with confidence intervals and slice guardrails. The release controller requires sample sufficiency, safety floors and operational budgets, binds approval to exact run evidence, then supports canary and rollback. Production traces enter privacy-aware quarantine, are redacted and reviewed before becoming versioned evaluation examples.
Open the complete worked design in a new tab →The new tab keeps this draft visible. Save it on this device before closing the page.