Learning pathsPricingChecking sign-in…
← ML curriculum

AISD-003 · DESIGN PRACTICE

Practice: Multi-tenant LLM inference gateway

Draft the complete decision system before revealing the guidance. Use a 45-minute rehearsal budget and adapt it to the interviewer’s direction.

Preserve eligibility, available-at-decision inputs, mature labels, bounded work and compatible releases. Explain alternatives and test a normal, boundary and failure case.

Your practice draft

Design a multi-tenant LLM inference gateway serving multiple model providers and GPU pools. Include tenant quotas, routing/fallback, streaming reconnect, cancellation, safe caching and accurate usage settlement. Explain how you preserve run identity when a provider or worker fails.

Saved locally on this device. The worked answer does not score your draft. Existing lesson notes and completion are separate.

Progressive hints

Hint 1

Start with tenant identity, immutable model bundle and a durable run/request key.

Hint 2

Separate admission and reservation from queueing and actual GPU execution.

Hint 3

Model prefill tokens, decode tokens, active sequences and KV memory as separate capacity constraints.

Hint 4

Route only among candidates that satisfy hard compatibility and tenant policy; fallback may change quality and price.

Hint 5

Persist ordered stream events before exposing them so reconnect replays instead of regenerating.

Hint 6

Treat cancellation as intent and settle usage only from terminal evidence; reconcile late reports exactly once.

Self-review evidence

For each criterion, record Missing, Prompted, Independent or Adaptive reasoning. No automated scoring or offer prediction is provided.

  • Tenant identity and resource authorization are derived server-side on every API.
  • Run idempotency binds tenant, request key and canonical payload hash.
  • Model aliases resolve to immutable compatible bundles pinned to the run.
  • Admission atomically reserves budget and checks context, deadline and capacity.
  • Routing separates hard constraints from live capacity/cost ranking.
  • Prefill, decode, active sequences and KV memory are independently bounded.
  • Streaming uses durable ordered events, owner epochs and explicit retention floors.
  • Cache reuse is scoped by tenant/trust domain, bundle, policy and freshness.
  • Cancellation and provider timeouts do not imply zero usage or safe immediate retry.
  • Usage settlement is evidence-based, idempotent and reconcilable.
  • Evaluation covers quality, latency, isolation, cost and recovery semantics.
Reveal worked design guidance

Authenticate the caller and derive tenant scope. Resolve the requested alias to an immutable model bundle, render/tokenize the full input, validate context and output caps, and atomically create the run plus tenant reservation under (tenant, request key, payload hash). The router filters candidates by entitlement, residency, modality, context, quality and tool compatibility, then ranks compatible pools by queue, prefill/decode headroom, KV occupancy, latency, errors and cost. A permitted fallback is revalidated and recorded. Workers emit sequenced events under a run owner epoch; persist before client delivery. Reconnect replays retained events. Cancellation stops new commits and signals the worker but keeps unresolved liability until terminal stop/usage evidence. Usage reports are ordered and deduplicated; settlement occurs once from pinned price and terminal evidence. Cache keys include bundle, tenant/trust namespace, policy, evidence and generation parameters. Dashboards track TTFT, inter-token latency, KV headroom, queue age, interrupted runs, replay expiry and cost per successful task.

Open the complete worked design in a new tab →

The new tab keeps this draft visible. Save it on this device before closing the page.