Learning pathsPricingChecking sign-in…
← ML curriculum

AIF-008 · FOUNDATION

Model routing, fallbacks and unit economics

A production model gateway routes requests using compatibility, policy, capacity, deadline, quality and budget constraints. Fallback is a new governed decision—not a blind retry—and cost comparisons must include tokenization, output length, retries, cache behavior and quality.

Author self-review; independent technical review pending. Deterministic conceptual fixture; no live provider calls, production credentials or billing.

Prerequisites and scoped previews

Prompt contracts, structured outputs and tool calling

Separate trusted instructions from untrusted data, validate structured outputs and keep tool authority in application code.

Streaming, cancellation and backpressure

Design resumable ordered generation streams, cancellation fencing, bounded buffers and usage settlement.

What, when, why and how

Resolve an immutable model bundle before dispatch

what
A model bundle pins provider/model ID, tokenizer, input/output limits, capability contract, price schedule, safety policy and version.
when
Use it whenever a request may be queued, retried, streamed or served by more than one provider.
why
Mutable aliases and mid-run configuration changes make latency, quality, safety and billing hard to reproduce.
how
Resolve a versioned bundle at admission and store its ID with the run, trace, usage and replay metadata.

Route only among eligible candidates

what
Routing first filters by capability, region, data residency, tenant policy, context length, modality and safety requirements, then scores eligible capacity.
when
Use it for multi-model gateways, regional serving pools and workload-specific model selection.
why
A cheaper or healthier provider is not a valid fallback if it cannot satisfy the request contract or data policy.
how
Apply hard eligibility gates before soft scoring for queue wait, p95 latency, quality tier, price, rate limits and current capacity.

Fallback differs from retry

what
Retry repeats an operation against the same logical target; fallback selects a different eligible target and may change model behavior.
when
Use retry for classified transient failures and fallback only when the alternate preserves required capabilities and policy.
why
An ambiguous timeout may mean the original request is still executing; blindly resubmitting can duplicate work and cost.
how
Classify failure phase, use idempotency and run linkage, enforce attempt budgets, and disclose or record model substitution.

1. Build an admission and routing decision

At admission, authenticate the tenant, validate request shape, estimate input tokens, reserve budget and resolve a model bundle. Hard gates remove candidates that violate data residency, tool/JSON capability, context length, modality, safety tier or tenant allowlists. Only then rank eligible providers by queue delay, tail latency, quality tier, cost and capacity.

Keep the routing policy explainable. Record eligible candidates, rejection reasons, chosen target and policy version. A route that cannot explain why it chose a provider is difficult to debug and audit.

VISUAL WALKTHROUGH

See the system in motion

Filter incompatible providers before ranking eligible capacity, latency and cost.

Process overviewAvailable even when interactive playback cannot load.
  1. Resolve immutable model bundle

  2. Filter policy, region and capability

  3. Check context and output constraints

  4. Rank eligible pools by capacity and SLO

  5. Pin route and dispatch

Loading interactive walkthrough…

2. Classify failures before retrying or falling back

Distinguish admission rejection, connection failure before dispatch, provider 429, provider 5xx, timeout after dispatch, stream interruption and invalid model output. A timeout after dispatch is ambiguous: the provider may still be generating. Query the original request if supported, otherwise use a bounded linked replacement policy and preserve the original run as uncertain.

Fallback must re-check model capabilities, data handling, region, context budget, safety policy, output schema and price ceiling. If the alternate cannot preserve the contract, fail explicitly or ask the caller to approve a degraded mode.

VISUAL WALKTHROUGH

See the system in motion

Classify provider failures and revalidate a fallback before creating a linked replacement.

Process overviewAvailable even when interactive playback cannot load.
  1. Detect and classify failure phase

  2. Determine whether execution is ambiguous

  3. Preserve original run and usage evidence

  4. Revalidate policy and capabilities

  5. Check remaining deadline and budget

  6. Create linked fallback only when allowed

Loading interactive walkthrough…

3. Compare total cost and quality, not headline price

Estimate input and output tokens separately. Include cached-input pricing, reasoning tokens where billable, tool calls, retries, failed attempts, embedding/reranking dependencies and the expected output length. Use measured workload distributions rather than a single average prompt.

A cheaper model can increase total cost if it needs longer prompts, more retries, more output tokens or human correction. Compare quality on a representative task set and slice by language, task type, tenant tier and safety-sensitive use. Track cost per successful task alongside cost per token.

4. Protect fairness and overload behavior

Use per-tenant concurrency and spend limits, bounded queues, deadlines, circuit breakers and load shedding. Reserve capacity for interactive traffic separately from batch evaluations where necessary. A circuit breaker should prevent repeated calls to a failing dependency but must not bypass hard policy gates.

Do not retry 429 or 5xx responses in synchronized bursts. Respect retry-after, use jitter, cap attempts and ensure the remaining deadline and budget can afford another attempt.

Diagrams and relationships

Policy-aware model routing and fallback

Hard policy and capability filters precede soft latency/cost scoring; fallback revalidates the full request contract.

Scroll horizontally on smaller screens, or enlarge the diagram.

Policy-aware model routing and fallbackHard policy and capability filters precede soft latency/cost scoring; fallback revalidates the full request contract. Authenticated request to Hard eligibility gates: policy + capability. Hard eligibility gates to Capacity + cost score: eligible only. Capacity + cost score to Pinned model bundle: select + pin. Pinned model bundle to Provider execution: dispatch. Provider execution to Fallback revalidation: classified failure.policy + capabilityeligible onlyselect + pindispatchclassified failureAuthenticated requestHard eligibility gatesCapacity + cost scorePinned model bundleProvider executionFallback revalidation
Policy-aware model routing and fallbackHard policy and capability filters precede soft latency/cost scoring; fallback revalidates the full request contract. Authenticated request to Hard eligibility gates: policy + capability. Hard eligibility gates to Capacity + cost score: eligible only. Capacity + cost score to Pinned model bundle: select + pin. Pinned model bundle to Provider execution: dispatch. Provider execution to Fallback revalidation: classified failure.policy + capabilityeligible onlyselect + pindispatchclassified failureAuthenticated requestHard eligibility gatesCapacity + cost scorePinned model bundleProvider executionFallback revalidation

Engineer, Senior and Staff follow-ups

These are teaching lenses. Each answer changes the scope of reasoning; they are not official employer leveling criteria.

Engineer: Why is fallback not just another retry?

Fallback may change provider, model, tokenizer, capability, policy or price. It must revalidate the request contract and preserve run linkage; a retry repeats the same logical target under a bounded attempt policy.

Engineer: Which signals should be hard routing gates?

Data residency, tenant/provider allowlist, modality, tool or structured-output support, context limit, safety requirements and region. Cost and latency are soft ranking signals only after eligibility.

Senior: How do you handle timeout after provider dispatch?

Treat it as ambiguous rather than assuming no work occurred. Query the original request or use an idempotent bounded replacement policy, preserve the original run and reconcile duplicate or late usage.

Senior: How do you compare model cost fairly?

Measure input/output and cached tokens, retries, tool and retrieval costs, failed attempts and correction burden. Compare cost per successful task with quality and safety on representative slices.

Staff: How do you prevent a low-cost fallback from violating product quality?

Define capability and quality floors, evaluate candidates against a versioned task set, monitor online slices, expose degraded-mode metadata and block substitution when the task contract cannot be met.

Staff: How should routing evolve under provider-wide overload?

Use admission control, tenant fairness, bounded queues, deadlines, circuit breakers and load shedding. Route only to policy-compatible capacity and avoid synchronized retries with retry-after and jitter.

Practice and self-review

Your practice draft

A provider times out after dispatch. The gateway has a second model that is 40% cheaper but has a smaller context window and weaker tool calling. Design a safe fallback policy.

Saved locally on this device. The worked answer does not score your draft. Existing lesson notes and completion are separate.

Progressive hints

Hint 1

Classify whether dispatch may have occurred and preserve the original run.

Hint 2

Revalidate context, tool, residency, safety and output schema compatibility.

Hint 3

Compare total cost per successful task and quality, not token price alone.

Reveal worked answer guidance

Keep the original attempt in an uncertain state and query its status if possible. Do not blindly resubmit. If replacement is allowed, estimate context and verify the alternate supports required tools, structured output, region, data policy and quality floor. If not, fail or ask for explicit degraded-mode approval. Pin the alternate bundle, link it to the original run, cap attempts/deadline/budget, reconcile late usage and compare cost per successful task with quality and safety slices.

Evidence to check in your answer

  • Ambiguous post-dispatch timeout is handled safely.
  • Hard policy/capability gates precede routing scores.
  • Fallback is linked, bounded and observable.
  • Cost and quality are evaluated per successful task.

For each item, record Missing, Prompted, Independent or Adaptive evidence. Adaptive means you can revise the reasoning under a changed constraint. Reading completion and readiness are separate.

Sources and verification scope

Original synthetic examples and reference calculations. Sources support narrowly scoped background; they do not establish production performance or independent chapter review.

SRE workbook: data processing pipelines ↗ · Pipeline SLOs and operational ownership; do not copy source workload numbers as course production results.

Hidden Technical Debt in Machine Learning Systems ↗ · System-boundary and lifecycle risks; source lessons motivate original examples.

vLLM automatic prefix caching ↗ · Prefix KV reuse and its prefill-specific benefit; does not establish tenant authorization or cross-tenant cache policy.

OpenTelemetry GenAI token metrics ↗ · Token usage counters versus per-operation histograms; use billable usage when both billable and consumed usage are available.

Primary background links checked during authoring on 2026-10-19. Arithmetic, temporal eligibility, maturity and split reference functions are checked by the bounded local fixture suite. SQL is explanatory, not a migrated production query. No GPU, external dataset, model API or training job is required.