Learning pathsPricingChecking sign-in…
← ML curriculum

AISD-003 · WORKED SYSTEM DESIGN

Multi-tenant LLM inference gateway

Design a multi-tenant inference gateway that exposes a stable, policy-governed API over changing model providers and serving pools. The gateway owns identity, tenant isolation, model resolution, admission, budgets, routing, streaming, cache scope, cancellation and evidence-based usage settlement. Provider capacity is not interchangeable: context compatibility, quality tier, prefill/decode throughput, KV memory, deadlines and survivor headroom all constrain safe routing.

independent technical review pending. source-contract checks pending.

Practice this design before reading →

Prerequisites and scoped previews

Events, identity and data contracts

Define keys, event/ingestion/availability times, schema evolution, dedupe and permitted use.

Calibration, uncertainty and decision thresholds

Distinguish score quality from calibrated probabilities, uncertainty and action capacity.

AI coding assistant with repository context and safe tool execution

Commit-pinned code retrieval, safe patch generation, isolated validation and approval-gated repository writes.

What, when, why and how

Treat the gateway as a control plane, not a proxy

what
The gateway authenticates callers, resolves immutable model bundles, applies policy, admits work and records authoritative run state before dispatching to serving pools.
when
When many products, tenants and providers share inference infrastructure.
why
A thin reverse proxy cannot enforce budgets, replay semantics, tenant cache isolation or safe provider fallback.
how
Separate request admission/control from data-plane token streaming, and persist the selected bundle, policy and reservation on each run.

Model compute demand in tokens and memory

what
Prompt prefill, output decode, active sequence slots and KV-cache memory are distinct capacity dimensions.
when
For variable-length prompts, long outputs, continuous batching and GPU-backed inference.
why
Requests per second and average token counts hide the bottleneck that drives queueing and tail latency.
how
Measure compatible pool throughput by bundle and workload distribution; reserve headroom for KV growth, long prompts and failure-domain loss.

Make tenant admission atomic

what
Admission checks tenant quota, concurrency, deadlines, context limits and pool capacity before accepting a run.
when
At high concurrency or when multiple services reserve the same tenant budget.
why
Separate check-then-increment calls allow concurrent requests to overspend or over-admit.
how
Create the run and its budget reservation in one conditional transaction or a recoverable reservation protocol with stable request identity.

Pin each run to a reproducible model bundle

what
A public model alias resolves to an immutable bundle: weights digest, tokenizer, template, adapter, runtime compatibility and policy tier.
when
Across retries, reconnects, provider rollouts and model alias changes.
why
Changing tokenizer or prompt template mid-run makes replay, token accounting and quality claims inconsistent.
how
Persist bundle digest and price/policy versions on the run; a fallback is a new explicit routing decision with compatibility and budget checks.

Separate visible stream commits from generation

what
Stream events have durable run identity, owner epoch and sequence; reconnect replays committed output rather than regenerating it.
when
For SSE/WebSocket streaming and flaky client networks.
why
A reconnect can otherwise duplicate text, create a second billable generation or splice incompatible output.
how
Commit ordered events before exposing them, fence stale workers and return an explicit retention boundary when old events expire.

Cache only within an authorized equivalence class

what
Prompt/KV and response caches reuse work only when model bundle, input, tenant trust scope, authorization, evidence and freshness are compatible.
when
For repeated prompts, retrieval-augmented answers or shared model prefixes.
why
Text equality does not imply permission to reuse another tenant's context or output.
how
Use canonical keys with policy/tenant namespace, bundle and evidence versions; reauthorize reads and measure side-channel leakage.

Settle usage from terminal evidence

what
Reservations, actual provider usage, visible output and billing adjustments are separate ledgers with explicit reconciliation.
when
For disconnects, cancellation, worker loss, retries and provider billing differences.
why
A socket close or cancellation request does not prove compute stopped or usage is zero.
how
Keep unresolved liability bounded, fence late reports, settle once from ordered terminal evidence and reconcile discrepancies with an audit trail.

Frame the inference gateway problem

Build one stable API for applications that need text generation, embeddings and optionally reranking, while the platform routes across model versions, providers and dedicated serving pools. The gateway must protect tenant boundaries, provide predictable latency and cost, and survive provider or worker failures without silently changing the meaning of an accepted request.

Synthetic interview assumptions: 100,000 active tenants, 5,000 peak concurrent requests, 100 new generation requests/second at the busy hour, 2,000 prompt tokens and 300 output tokens on average. This is 200,000 prefill tokens/second and 30,000 decode tokens/second before burst factors. These are exercise inputs, not measured production capacity. Long-context and long-output tails must be modeled separately.

Clarify requirements, SLOs and product promises

Functional requirements: authenticate a caller, resolve a model alias, validate context/output limits, reserve tenant budget, select a compatible pool, stream output, support cancellation and reconnect, record usage and expose per-tenant analytics. Optional capabilities include embeddings, reranking, structured output and tool-call payloads; they should share policy primitives without pretending their compute profiles are identical.

Define SLOs separately: admission latency, time-to-first-token (TTFT), inter-token latency, end-to-end completion, stream availability and usage reconciliation delay. Define quality-tier guarantees, maximum queue wait, output truncation behavior, data retention and whether fallback is allowed. Never claim a single 'inference latency' metric for prefill, decode and queueing.

Establish tenant identity and trust boundaries

Derive tenant and principal from authenticated credentials at the edge. A request body may select a logical model alias and provide a request key, but it cannot choose its tenant, override plan limits or request another tenant's run. Authorize run status, event replay, cancellation, cache access and usage queries on every call.

Separate control-plane secrets and provider credentials from the data plane. Use tenant-scoped API keys or OAuth clients, short-lived provider credentials where possible, encryption in transit/at rest, regional residency policy, per-tenant retention and redaction before logs. A shared GPU does not imply shared cache trust: prefix/KV reuse and response-cache hits need explicit namespace and authorization checks.

VISUAL WALKTHROUGH

See the system in motion

Trace authentication, tenant derivation, request identity and durable admission.

Process overviewAvailable even when interactive playback cannot load.
  1. Authenticate the caller.

  2. Derive tenant scope from trusted credentials.

  3. Bind request key to canonical payload hash.

  4. Check policy, limits and deadline.

  5. Persist the accepted run.

Loading interactive walkthrough…

Define a stable API and run lifecycle

A generation request includes model alias, canonical messages/input, output cap, optional deadline, response format, request key and stream preference. Return a run ID and accepted status only after the admission record and reservation are durable. GET run/events?after=sequence replays retained events; DELETE run records cancellation intent; GET usage returns reconciled usage with a stated freshness watermark.

Bind idempotency to (tenant, request_key) and a canonical request hash. The same key and same hash resolves to the original run; the same key with different content is a conflict. A new generation after worker loss is a new run linked to the interrupted run, not a continuation that reuses its event cursor.

APIPurposeImportant contract
POST /v1/inferenceAdmit and start a runDurable run ID, pinned bundle and reservation
GET /v1/runs/{id}Read run stateTenant/resource authorization
GET /v1/runs/{id}/events?after=NReplay outputOrdered sequence and retention floor
DELETE /v1/runs/{id}Cancel workCancellation intent, not proof of stop
GET /v1/usageRead accountingSettled, reserved and unresolved amounts separated

Build a versioned model catalog

The catalog maps friendly aliases such as 'fast-chat' or 'reasoning-large' to allowed immutable serving bundles. A bundle pins provider/model ID, weights or provider revision where available, tokenizer, system template, adapter/LoRA identity, context limit, output cap, modality, runtime/precision compatibility, region, quality tier and price schedule.

Resolve the alias once at admission and persist the resulting bundle digest. A retry or reconnect for the same run must not silently resolve a newer alias target. Keep a compatibility matrix for fallback candidates; equal API shape does not imply equal context window, tool support, safety behavior, region or quality.

Validate input and admit work before dispatch

Render the final model input first, including system instructions, conversation history, retrieved evidence, tool schemas and output-format constraints. Tokenize using the selected bundle's tokenizer. Client token estimates are advisory. Require input tokens plus reserved maximum output to fit context limits and per-tenant policy; explicitly truncate according to a documented policy or reject with a useful error.

Admission also checks deadline feasibility, queue age, tenant concurrency, token/cost budget, model tier, region and compatible pool capacity. Do not accept work into an unbounded queue. The accepted response means the run and reservation are durable; it does not mean a GPU has started decoding.

Reserve tenant budgets atomically

Choose the quota unit deliberately: prompt/output tokens, money in integer micro-units, requests, concurrent runs or compute credits are not interchangeable. For a token contract, reserve input tokens plus maximum output tokens. With 10,000 units available, a 9,000-token prompt and 2,000 output cap cannot be admitted; a smaller cap may fit, but still needs queue and KV admission.

Create the run and reservation atomically where possible, conditioned on the same tenant balance and stable request identity. If quota and worker admission are separate services, model provisional reservation, start authorization, expiry and compensation explicitly. A lost acceptance response must resolve the same run instead of reserving twice.

Reservation fieldExampleWhy it matters
tenant_idderived from authNever trust request body tenant
request_key + payload_hashclient retry identityPrevents duplicate admission
bundle_digestimmutable model bundlePins tokenizer and compatibility
reserved_unitsinput + output capBounds maximum liability
price_versionversioned rate cardReproducible settlement
statereserved / started / settled / unresolvedSupports recovery and audit

VISUAL WALKTHROUGH

See the system in motion

Follow canonical tokenization into an atomic tenant budget reservation and durable dispatch intent.

Process overviewAvailable even when interactive playback cannot load.
  1. Render the complete model input.

  2. Tokenize with the pinned bundle.

  3. Conditionally reserve tenant budget.

  4. Commit run and reservation.

  5. Publish dispatch intent.

Loading interactive walkthrough…

Route by policy, capability and live capacity

The router first filters by hard constraints: tenant entitlement, data residency, modality, context length, tool/structured-output support, safety tier, bundle compatibility and provider allowlist. It then ranks eligible pools using queue delay, prefill/decode headroom, KV occupancy, recent TTFT/inter-token latency, error rate, cost and locality.

Fallback is not a blind redirect. A smaller or alternate model may change quality, tokenizer, context limits, tool semantics, region and price. Revalidate the fully rendered prompt and reserved budget against the candidate. Pin the chosen candidate and record why it was selected. If no candidate satisfies hard constraints and deadline, reject or queue within a bounded policy rather than silently violating the contract.

VISUAL WALKTHROUGH

See the system in motion

Filter model pools by hard constraints before ranking live capacity and routing.

Process overviewAvailable even when interactive playback cannot load.
  1. Resolve immutable model bundle.

  2. Filter entitlement, region and capability.

  3. Read queue, prefill, decode and KV signals.

  4. Rank compatible pools.

  5. Record route and dispatch.

Loading interactive walkthrough…

Isolate serving pools and schedule fairly

Partition pools by model bundle, hardware/runtime compatibility, region and trust tier. A worker with spare memory but incompatible weights is not spare capacity. Use bounded per-tenant queues and weighted fair scheduling so one noisy tenant cannot monopolize the fleet. Apply concurrency caps and priority classes with starvation controls.

Track cold-load time, warm replicas, autoscaling delay, queue age and failure-domain placement. Keep control-plane admission responsive even when a provider is unhealthy. Circuit breakers should stop routing to failing pools, but half-open probes and gradual recovery prevent a thundering herd.

Optimize prefill, decode and KV-cache capacity

Prefill processes prompt tokens and often creates bursty compute demand; decode produces output tokens over time and is sensitive to inter-token latency. Continuous batching improves utilization but can make long prompts interfere with interactive decode. Chunked prefill, admission shaping and scheduling weights balance TTFT and inter-token SLOs.

KV cache consumes memory as sequences grow. Estimate resident-token capacity from measured per-token KV allocation, model layers/heads, precision, parallelism and runtime overhead. Reserve for output growth and fragmentation; do not admit to the theoretical physical-memory ceiling. Preemption or eviction needs an explicit policy and a visible effect on latency/recomputation.

VISUAL WALKTHROUGH

See the system in motion

Visualize independent prefill, decode, sequence-slot and KV-memory constraints.

Process overviewAvailable even when interactive playback cannot load.
  1. Estimate prompt and output demand.

  2. Schedule bounded prefill chunks.

  3. Batch compatible decode sequences.

  4. Reserve KV growth and sequence slots.

  5. Emit tokens under latency and memory limits.

Loading interactive walkthrough…

Make streaming ordered, resumable and tenant-safe

Assign every run an owner epoch and monotonically increasing event sequence. Persist each visible event before sending it to the client if the product promises replay. A duplicate identical event resolves to the same record; conflicting bytes at the same sequence are rejected. Terminal events close the epoch and late worker output cannot extend it.

If a connection drops after sequence 35 and committed events 36–40 remain retained, reconnect replays 36–40 without invoking the model again. If the requested sequence is older than the retention floor, return an explicit resume-expired response and a snapshot/completeness boundary. A stale worker must fail the durable append authority check after ownership changes; an in-memory gateway buffer is not the source of truth.

VISUAL WALKTHROUGH

See the system in motion

Replay committed stream events after reconnect without starting a second generation.

Process overviewAvailable even when interactive playback cannot load.
  1. Generate output delta.

  2. Append with owner epoch and sequence.

  3. Commit before client delivery.

  4. Authorize reconnect cursor.

  5. Replay retained events in order.

Loading interactive walkthrough…

Design safe prefix, KV and response caching

Prefix/KV caching can avoid repeated prefill only when the complete token prefix, tokenizer, model bundle, position/attention state, adapter and runtime compatibility match. Namespace by tenant or approved trust domain and enforce authorization before reuse. Textually identical prompts from two tenants are not sufficient proof of cache equivalence.

Response caching needs a canonical key over normalized input, model bundle, system/template version, generation parameters, retrieval evidence/version, safety policy and tenant visibility scope. Recheck access and freshness on every hit. Avoid caching sensitive or highly dynamic outputs by default. Measure hit rate, stale-hit rate, isolation failures and timing side channels; cache invalidation is part of data governance.

Handle provider failures without duplicate work

Classify errors: validation and policy denials are terminal; rate limits may retry after provider guidance; transient network failures may retry only when execution status is known; overload should trigger admission shedding or eligible routing. Use bounded exponential backoff with jitter, per-provider concurrency limits and circuit breakers.

An ambiguous timeout after dispatch is not proof that the provider did not execute. Query provider operation status or retain an unresolved attempt and liability before retrying. Never append output from a second generation to the first run's stream. If product policy allows replacement, create a linked run with a new identity, explicit budget and user-visible interruption state.

VISUAL WALKTHROUGH

See the system in motion

Distinguish known failures from ambiguous execution before considering a compatible fallback.

Process overviewAvailable even when interactive playback cannot load.
  1. Detect provider failure.

  2. Classify execution certainty.

  3. Preserve original run evidence.

  4. Revalidate policy, context and budget.

  5. Create a linked replacement only when allowed.

Loading interactive walkthrough…

Cancel safely and settle usage from evidence

Cancellation records durable intent, stops new output commits, signals the serving worker and fences further writes. It does not prove that a partitioned worker stopped consuming GPU. Do not immediately release the full reservation or start unlimited replacement work; retain bounded unresolved liability until stop and usage evidence arrive.

Persist cumulative usage reports ordered by owner epoch and report sequence. Ignore identical replay, reject conflicting or regressing reports and settle once from terminal evidence. Pin the price version. Provider billing, platform quota and delivered output may differ; reconcile them with auditable adjustments rather than silently mutating a closed charge.

VISUAL WALKTHROUGH

See the system in motion

Follow cancellation intent, terminal usage evidence and one-time reservation settlement.

Process overviewAvailable even when interactive playback cannot load.
  1. Persist cancellation intent.

  2. Signal worker and fence output.

  3. Collect ordered terminal usage.

  4. Reconcile duplicate and late reports.

  5. Settle once and release remainder.

Loading interactive walkthrough…

Instrument the entire inference lifecycle

Trace admission, tokenization, reservation, queue wait, routing decision, provider dispatch, TTFT, each output chunk, cancellation, terminal usage and settlement. Attach tenant-safe dimensions such as tier, region, bundle digest and provider; never put raw prompts, secrets or personal data into high-cardinality metrics.

Dashboards should separate admission latency, queue age, TTFT, inter-token latency, completion, prefill/decode utilization, KV headroom, active sequences, cache hit/miss, provider errors, interrupted runs, replay expiry, unresolved reservation age and cost per successful task. Alert on invariant violations such as cross-tenant reads, duplicate settlement or stale-worker stream commits.

Estimate capacity and unit economics

At the exercise baseline of 100 requests/second with 2,000 prompt and 300 output tokens, average demand is 200,000 prefill and 30,000 decode tokens/second. Add peak factors, model-specific throughput, long-context distributions, batching efficiency, replica redundancy and failure-domain headroom. Request rate alone is not a capacity plan.

For an illustrative memory calculation, assume 80 GiB physical memory, 40 GiB for weights/runtime and 8 GiB safety reserve, leaving 32 GiB for KV. At a measured 256 KiB per resident token, the ideal ceiling is 131,072 resident tokens before fragmentation and other overhead. This is an exercise assumption, not a universal GPU figure. Track cost per successful request and per million input/output tokens, including idle warm capacity, failed attempts, retries, cache discounts and reconciliation.

Evaluate routing quality and roll out safely

Evaluate task quality and serving behavior together. Build a versioned suite across prompt lengths, output lengths, languages, tool calls, structured output, tenant isolation, cancellation, reconnect, provider overload and fallback. Score correctness, policy compliance, TTFT, inter-token latency, cost and recovery semantics. A fast but semantically degraded fallback is not a successful route.

Roll out a model bundle or router policy through offline replay, shadow decisions, canary tenants and gradual traffic shifts. Compare actual selected provider/model to the intended experiment. Rollback changes new admissions; it must not reinterpret the tokenizer, price, stream history or policy version of already accepted runs.

Walk through overload, disconnect and worker loss

A popular model pool overloads while a compatible smaller pool has headroom. The router checks tenant policy and quality downgrade permission, estimates queue/deadline feasibility and only then selects the smaller bundle with a recorded routing reason. If downgrade is forbidden, the request is rejected or bounded-queued rather than silently changing quality.

A client disconnects after receiving event 35. On reconnect, the gateway authorizes the run and replays committed 36–40. If the worker dies before terminal usage, mark the run interrupted, preserve visible output and retain unresolved reservation. A replacement is a new linked run. When delayed provider usage arrives, reconcile against the old run without allowing a stale owner to append output or settle twice.

Complete multi-tenant inference gateway architecture

The complete request path is authenticated application → gateway/API → tenant policy and model catalog → tokenizer/admission → atomic reservation → router/scheduler → compatible serving pool → durable sequenced stream → client. Cache, usage ledger, cancellation, observability and reconciliation are connected control paths with their own authority checks.

The gateway is authoritative for tenant identity, run lifecycle, stream sequence, reservation and settlement state. The serving worker is authoritative for measured execution and terminal usage evidence. The provider is authoritative for its own execution/billing records. Reconciliation joins these sources without allowing one to silently overwrite another.

VISUAL WALKTHROUGH

See the system in motion

Trace the complete multi-tenant inference path from authentication and atomic admission through serving, replay and settlement.

Process overviewAvailable even when interactive playback cannot load.
  1. Authenticate and derive tenant scope.

  2. Resolve immutable model bundle.

  3. Validate policy and request identity.

  4. Tokenize input and reserve budget.

  5. Check deadline and compatible capacity.

  6. Route to eligible serving pool.

  7. Execute prefill and decode under KV limits.

  8. Commit ordered stream events and replay.

  9. Collect usage evidence and settle once.

  10. Apply scoped cache and return outcome.

Loading interactive walkthrough…

Diagrams and relationships

Control plane and inference data plane

Admission and policy are durable control decisions; token streaming is a separately scaled data path.

Scroll horizontally on smaller screens, or enlarge the diagram.

Control plane and inference data planeAdmission and policy are durable control decisions; token streaming is a separately scaled data path. Application to Inference gateway: request. Inference gateway to Tenant policy: authorize. Inference gateway to Model catalog: resolve bundle. Tenant policy to Router + scheduler: admit. Model catalog to Router + scheduler: compatible bundle. Router + scheduler to Serving pool: dispatch.requestauthorizeresolve bundleadmitcompatible bundledispatchApplicationInference gatewayTenant policyModel catalogRouter + schedulerServing pool
Control plane and inference data planeAdmission and policy are durable control decisions; token streaming is a separately scaled data path. Application to Inference gateway: request. Inference gateway to Tenant policy: authorize. Inference gateway to Model catalog: resolve bundle. Tenant policy to Router + scheduler: admit. Model catalog to Router + scheduler: compatible bundle. Router + scheduler to Serving pool: dispatch.requestauthorizeresolve bundleadmitcompatible bundledispatchApplicationInference gatewayTenant policyModel catalogRouter + schedulerServing pool
Durable streaming and usage settlement

A reconnect replays committed output; accounting closes only from terminal evidence.

Scroll horizontally on smaller screens, or enlarge the diagram.

Durable streaming and usage settlementA reconnect replays committed output; accounting closes only from terminal evidence. Serving worker to Stream log: sequenced events. Stream log to Client replay: replay. Serving worker to Usage ledger: terminal report. Usage ledger to Reconciler: evidence. Reconciler to Tenant usage: settle once.sequenced eventsreplayterminal reportevidencesettle onceServing workerStream logClient replayUsage ledgerReconcilerTenant usage
Durable streaming and usage settlementA reconnect replays committed output; accounting closes only from terminal evidence. Serving worker to Stream log: sequenced events. Stream log to Client replay: replay. Serving worker to Usage ledger: terminal report. Usage ledger to Reconciler: evidence. Reconciler to Tenant usage: settle once.sequenced eventsreplayterminal reportevidencesettle onceServing workerStream logClient replayUsage ledgerReconcilerTenant usage
Admission and capacity decision

A request must fit tenant policy, context, deadline, budget and a compatible pool.

Scroll horizontally on smaller screens, or enlarge the diagram.

Admission and capacity decisionA request must fit tenant policy, context, deadline, budget and a compatible pool. Rendered request to Bundle tokenizer: canonical input. Bundle tokenizer to Budget reserve: token count. Budget reserve to Pool capacity: reserved run. Pool capacity to Admit / reject: deadline + KV.canonical inputtoken countreserved rundeadline + KVRendered requestBundle tokenizerBudget reservePool capacityAdmit / reject
Admission and capacity decisionA request must fit tenant policy, context, deadline, budget and a compatible pool. Rendered request to Bundle tokenizer: canonical input. Bundle tokenizer to Budget reserve: token count. Budget reserve to Pool capacity: reserved run. Pool capacity to Admit / reject: deadline + KV.canonical inputtoken countreserved rundeadline + KVRendered requestBundle tokenizerBudget reservePool capacityAdmit / reject

Engineer, Senior and Staff follow-ups

These are teaching lenses. Each answer changes the scope of reasoning; they are not official employer leveling criteria.

Engineer: Why do you track prefill and decode separately?

Prefill processes the prompt and drives TTFT; decode emits tokens over time and drives inter-token latency. Their compute profiles and scheduling interference differ, so one aggregate tokens-per-second number hides bottlenecks.

Engineer: What makes an inference request idempotent?

Scope a request key to the authenticated tenant and bind it to a canonical payload hash. Identical retries resolve the original run; changed payload under the same key conflicts.

Senior: How do you route around an overloaded model pool?

Filter by hard constraints first—tenant entitlement, residency, context, modality, quality and tool support—then rank compatible pools by queue delay, prefill/decode headroom, KV occupancy, latency, errors and cost. A downgrade requires explicit policy and a fresh budget/context check.

Senior: How do you support reconnect without generating twice?

Persist sequenced stream events before exposing them. Authorize the run, replay retained events after the cursor, and return a clear retention boundary when history has expired. Never treat a second generation as continuation of the original stream.

Senior: Why can't you release budget immediately on cancellation?

Cancellation is intent, not proof that a partitioned worker stopped. Keep bounded unresolved liability until terminal stop and usage evidence arrive; reconcile late reports against the original run.

Staff: How do you prevent cross-tenant cache leakage?

Namespace caches by tenant or explicitly approved trust domain, pin bundle and policy/evidence versions, authorize each hit, avoid unsafe output caching, and test timing/content side channels. Text equality alone is not cache authorization.

Staff: How would you build a reliable usage ledger across providers?

Separate reservation, provider-reported usage, platform quota and invoice settlement. Order reports by run/owner epoch and report sequence, deduplicate identical replay, reject conflicts, settle once from terminal evidence and reconcile discrepancies through audited adjustments.

Staff: What does a safe model/provider rollout mean for in-flight runs?

New admissions can move to a new bundle after canary checks, but accepted runs retain their pinned tokenizer, template, policy, price and stream ownership. Rollback affects new runs; in-flight state is migrated only through an explicit compatibility contract.

Practice and self-review

Your practice draft

Design a multi-tenant LLM inference gateway serving multiple model providers and GPU pools. Include tenant quotas, routing/fallback, streaming reconnect, cancellation, safe caching and accurate usage settlement. Explain how you preserve run identity when a provider or worker fails.

Saved locally on this device. The worked answer does not score your draft. Existing lesson notes and completion are separate.

Progressive hints

Hint 1

Start with tenant identity, immutable model bundle and a durable run/request key.

Hint 2

Separate admission and reservation from queueing and actual GPU execution.

Hint 3

Model prefill tokens, decode tokens, active sequences and KV memory as separate capacity constraints.

Hint 4

Route only among candidates that satisfy hard compatibility and tenant policy; fallback may change quality and price.

Hint 5

Persist ordered stream events before exposing them so reconnect replays instead of regenerating.

Hint 6

Treat cancellation as intent and settle usage only from terminal evidence; reconcile late reports exactly once.

Reveal worked answer guidance

Authenticate the caller and derive tenant scope. Resolve the requested alias to an immutable model bundle, render/tokenize the full input, validate context and output caps, and atomically create the run plus tenant reservation under (tenant, request key, payload hash). The router filters candidates by entitlement, residency, modality, context, quality and tool compatibility, then ranks compatible pools by queue, prefill/decode headroom, KV occupancy, latency, errors and cost. A permitted fallback is revalidated and recorded. Workers emit sequenced events under a run owner epoch; persist before client delivery. Reconnect replays retained events. Cancellation stops new commits and signals the worker but keeps unresolved liability until terminal stop/usage evidence. Usage reports are ordered and deduplicated; settlement occurs once from pinned price and terminal evidence. Cache keys include bundle, tenant/trust namespace, policy, evidence and generation parameters. Dashboards track TTFT, inter-token latency, KV headroom, queue age, interrupted runs, replay expiry and cost per successful task.

Evidence to check in your answer

  • Tenant identity and resource authorization are derived server-side on every API.
  • Run idempotency binds tenant, request key and canonical payload hash.
  • Model aliases resolve to immutable compatible bundles pinned to the run.
  • Admission atomically reserves budget and checks context, deadline and capacity.
  • Routing separates hard constraints from live capacity/cost ranking.
  • Prefill, decode, active sequences and KV memory are independently bounded.
  • Streaming uses durable ordered events, owner epochs and explicit retention floors.
  • Cache reuse is scoped by tenant/trust domain, bundle, policy and freshness.
  • Cancellation and provider timeouts do not imply zero usage or safe immediate retry.
  • Usage settlement is evidence-based, idempotent and reconcilable.
  • Evaluation covers quality, latency, isolation, cost and recovery semantics.

For each item, record Missing, Prompted, Independent or Adaptive evidence. Adaptive means you can revise the reasoning under a changed constraint. Reading completion and readiness are separate.

Sources and verification scope

Original synthetic examples and reference calculations. Sources support narrowly scoped background; they do not establish production performance or independent chapter review.

Primary background links checked during authoring on 2026-10-19. Arithmetic, temporal eligibility, maturity and split reference functions are checked by the bounded local fixture suite. SQL is explanatory, not a migrated production query. No GPU, external dataset, model API or training job is required.