LLM Inference Platform
Serve multiple model versions under tenant cost, token, latency and GPU-memory budgets with bounded admission and streaming recovery semantics.
Serve multiple model versions under tenant cost, token, latency and GPU-memory budgets with bounded admission and streaming recovery semantics. This worked design targets SDE-3; the scale figures below are exercise assumptions, not claims about a named company's implementation.
Clarify the prompt
| Ask the interviewer | Decision to record |
|---|---|
| Which models, context lengths and quality tiers? | Request count alone does not express compute demand. |
| What are time-to-first-token and inter-token targets? | Streaming throughput differs from admission latency. |
| May the router downgrade models or cache outputs? | Require explicit quality, privacy and freshness policies. |
Requirements and scope
The core deliverable is: Serve multiple model versions under tenant cost, token, latency and GPU-memory budgets with bounded admission and streaming recovery semantics. The first design implements the APIs below and the state constraints in the data model. Agree the latency, freshness, retention and failure policy before adding infrastructure. Every successful acknowledgment must have an explicit meaning: accepted intent, committed state or completed external work. These are separate milestones.
Capacity and bottleneck reasoning
Assume 100 requests/s with 2,000 prompt tokens and 300 output tokens on average: 200,000 prefill tokens/s and 30,000 decode tokens/s. Capacity comes from benchmarked per-model throughput and KV memory under the actual length distribution, with headroom for long tails.
Keep average and peak demand separate. Add replication, indexes, retries and safety headroom to raw arithmetic before selecting capacity. Load-test the busiest ownership key and recovery traffic as well as uniform happy-path throughput.
APIs and data model
- POST /inference {model,input,maxOutputTokens,requestKey,deadline} -> runId/SSE
- GET /runs/{id}/events?after=... -> retained stream events
- DELETE /runs/{id} -> cancellation intent; GET /usage -> reconciled usage
model(version,tokenizer,context_limit,worker_pool,quality_tier)
run(run_id PK,tenant,request_key,payload_hash,model_version,status,deadline)
UNIQUE(tenant,request_key); authorize run lookup by tenant
budget(tenant,reserved_tokens,used_tokens); stream(run_id,sequence,delta,expires_at)Authorize resource ownership at the API boundary; do not trust IDs or state supplied by a client. Bind retry identity to the caller and payload for mutating operations. Reject an incompatible replay instead of returning an unrelated stored result.
Component relationships
Read the diagram
Follow the labeled relationship panels; they are separate directed paths, not one global timeline.
Step-by-step design
1. Establish the authority
Authenticate and tokenize before admission. Reserve tenant token/cost budgets atomically, enforce context/output limits and reject work that cannot meet its queue deadline. Route only among permitted quality tiers and model versions; retain that decision on the run.
2. Make retries and concurrent work safe
Benchmark and isolate model pools. Continuous batching improves utilization but long prefills can hurt decode latency; schedule under both token and KV-memory budgets. Reuse compatible prefixes only within permitted trust scopes and account for cache misses.
3. Scale and recover the critical path
Stream sequenced durable events where replay is required. Cancellation stops future work and releases unused reservations, but consumed compute still counts. On worker failure, report an interrupted run; silently regenerating a partial response can change output and double cost.
Worked request and concurrency example
Tenant A has a 10,000-token budget remaining and submits a 9,000-token prompt with 2,000 maximum output. Admission rejects or requests a smaller output cap before occupying a GPU. An accepted smaller run pins model v7 and emits events 1-40. Reconnect after event 35 replays retained 36-40 rather than generating again. If its worker dies, a separately authorized replacement run receives a new identity and the usage ledger reconciles actual consumed tokens.
Failure and recovery walkthrough
A popular model pool overloads while another is idle. Queue-time limits, admission shedding and a permitted smaller-model fallback prevent unbounded waiting; unused capacity in an incompatible model pool cannot be treated as interchangeable without considering loading time and quality.
Read the diagram
This numbered trace follows one scenario; explain the durable evidence at each transition.
Decisions and alternatives
| Decision | Trade-off |
|---|---|
| Alternative 1 | Dedicated pools isolate model SLOs but can waste capacity. Shared pools improve utilization while introducing load/unload overhead and noisy neighbors. |
| Alternative 2 | Prefix/KV caching saves compatible prefill work; it is different from reusing a completed answer. Completed-output caches require stronger semantic, permission and freshness checks. |
Operational contract
Track queue delay, TTFT, inter-token latency, prefill/decode throughput, KV occupancy, cancellation lag and cost per completed run. Canary model/runtime pairs with quality and latency guardrails and test worker loss mid-stream.
Choose alert thresholds from the agreed user-facing objective. Assign an owner for stuck work and a replay/repair procedure that preserves the original business identity. Bound queue depth, concurrency and recovery throughput so an outage does not produce an uncontrolled catch-up burst.
Foundation, senior and staff depth
- Foundation: explain the user action, model, first complete request and the correctness constraint without relying on a list of technologies.
- Senior: defend this lesson's partition/coordination boundary, capacity estimate and failure recovery; explain the cost of the rejected alternative.
- Staff: identify failure domains and operational ownership, stage a rollout with measured guardrails, and state when rollback stops being safe.
Practice before revealing the answer
Spend 45 minutes: 5 clarifying scope, 7 on capacity/APIs/state, 12 on the first architecture, 15 on the concurrency and failure examples, and 6 on trade-offs and operations. Draw your own design before comparing it with this chapter.
Question: Why can a system meet tokens/second throughput while missing latency targets?
Show answer and explanation
Answer: Large batches and long prefills can increase queueing, first-token and inter-token delays. Measure each stage and schedule by token/memory workload and deadline; aggregate throughput hides per-request tail latency.
Verification and rollout
Reproduce the worked example using two concurrent actors or a repeated request where applicable. Inject the stated failure between the durable transition and its acknowledgment; verify the resulting state and user-visible outcome. For read/ML paths, test missing or stale dependencies and verify the documented fallback still enforces permissions. Use a small canary, compare correctness and user-facing latency with the baseline, and roll back when those guardrails regress. Do not use an average throughput number as evidence that a tail-latency target is met.
Primary technical reference
Official documentation for the mechanism used in this exercise. This lesson is an original interview design, not a description of undisclosed company internals.
Related study
Before this design, review these mechanisms and identify which owns the critical state transition:
Use the Interview Playbook to adapt the depth and timing, then return to the question breakdown overview.
Why can a system meet tokens/second throughput while missing latency targets?
Your design draft
Clarify assumptions, explain your approach, and test the difficult cases. Save your draft, then compare it with the study notes.
Self-review checklist
Self-guided practice. Automated AI feedback and code execution are not connected.