Learning pathsPricingChecking sign-in…
← ML curriculum

AISD-005 · WORKED SYSTEM DESIGN

Real-time voice assistant

Design a low-latency, interruptible, multi-tenant voice assistant that streams microphone audio, transcribes incrementally, manages conversational turns, invokes policy-scoped tools, synthesizes speech and recovers cleanly from network and provider failures.

independent technical review pending. source-contract checks pending.

Practice this design before reading →

Prerequisites and scoped previews

Leakage, splits and evaluation contamination

Prevent future information, entity overlap, preprocessing leakage and test-set reuse.

Deep models and sequence context

Justify neural/sequence architectures from data/task structure and resource constraints.

Scoped prerequisite preview. The complete chapter is planned; this preview does not count as authored coverage.

Regression and forecasting evaluation

Match MAE/RMSE/quantile/interval metrics to horizon and error costs; backtest over time.

Scoped prerequisite preview. The complete chapter is planned; this preview does not count as authored coverage.

Multi-tenant LLM inference gateway

Tenant-safe model routing, atomic budget admission, KV-aware serving, resumable streams and evidence-based usage settlement.

Multimodal video understanding platform

Design temporal video/audio ingestion, multimodal retrieval and grounded timestamped answers.

What, when, why and how

Treat voice as a real-time interaction, not chat with a microphone

what
Voice has continuous media transport, partial hypotheses, turn boundaries, audio rendering and interruption.
when
For conversational assistants, contact centers and hands-free interfaces.
why
A request/response chat architecture adds latency and cannot handle overlapping speech or barge-in.
how
Model media sessions, turns and utterance segments separately; use streaming transport and explicit deadlines.

Budget latency across the whole turn

what
Perceived latency is the sum of capture, transport, endpointing, ASR, orchestration, model first-token, TTS first-audio and playback buffers.
when
When users expect natural conversation.
why
Optimizing only model inference leaves endpointing or audio buffering as the dominant delay.
how
Define stage budgets and measure p50/p95 time-to-first-audio and end-of-turn completion.

Separate partial hypotheses from committed transcript

what
Streaming ASR revises provisional text until a stable segment is committed.
when
For low-latency turn-taking and live captions.
why
Acting on unstable words can trigger wrong tools or responses.
how
Track hypothesis revisions, stability and finalization; permit low-risk speculative work but gate side effects on committed intent.

Make turn ownership explicit

what
A turn manager owns turn IDs, endpointing, interruption, cancellation and output state.
when
For duplex audio where user speech can overlap assistant speech.
why
Without explicit ownership, stale LLM/TTS chunks can play after the user interrupts.
how
Use monotonically increasing turn generations and fence every downstream result against the active turn.

Stream TTS with playback-aware backpressure

what
TTS can emit audio chunks before the complete response is ready.
when
When minimizing time to first audio.
why
Unbounded synthesis or playback buffers increase latency and make interruption sluggish.
how
Bound queued audio duration, propagate cancellation and keep text/audio chunk alignment metadata.

Keep tools behind deterministic policy gates

what
The model proposes structured actions; an authorization layer validates identity, arguments, risk and confirmation.
when
When voice can access accounts or perform actions.
why
Speech recognition errors and prompt injection can produce unintended side effects.
how
Require committed transcript, schema validation, least privilege, idempotency and human confirmation for high-impact actions.

Design graceful degradation across providers

what
ASR, LLM and TTS are separate failure domains with different fallback options.
when
For availability-sensitive voice sessions.
why
Retrying the entire turn can duplicate actions and discard already-produced audio.
how
Persist turn state, retry only safe stages, switch compatible providers and clearly signal degraded mode.

Frame the real-time voice assistant

A user speaks naturally, may pause or interrupt, and expects the assistant to respond in speech. The system must stream microphone audio, produce provisional and final transcripts, decide when a turn is complete, generate a response, optionally call tools and stream synthesized audio back while remaining interruptible.

Synthetic exercise assumptions: 100,000 daily active users, 8 minutes average voice time/day, 5 concurrent sessions per 1,000 DAU at peak, 16 kHz mono input and 24 kHz output. These are interview assumptions, not production measurements. Calculate concurrency from active session minutes and peak factor, not DAU alone.

Define interaction requirements and SLOs

Functional requirements: session establishment, streaming audio ingress/egress, partial/final ASR, endpointing, turn management, barge-in, conversational context, safe tools, streaming TTS, reconnect and session termination. Non-functional requirements: low time-to-first-audio, bounded end-of-turn latency, intelligibility, interruption responsiveness, availability, privacy and cost.

Track stage-level p50/p95/p99, time-to-first-transcript, endpointing delay, time-to-first-model-token, time-to-first-audio, total turn latency, barge-in-to-audio-stop, dropped audio, jitter-buffer underruns and task success. Set product-specific targets after user research; do not assert universal latency thresholds.

Choose the media transport and session contract

For browser/mobile interactive audio, WebRTC is usually a strong choice when low-latency duplex media, jitter handling and network adaptation matter. WebSocket can be appropriate for controlled environments, simpler audio-frame streaming or text/control channels, but the application must explicitly handle framing, pacing, loss and congestion. SIP/PSTN gateways are a separate ingress boundary for telephony.

A session record pins tenant, authenticated actor, consent state, codec/sample rates, locale, provider/model policy, expiry and session generation. Media credentials are short-lived and scoped. Separate the reliable control channel (session, turn, tool status) from real-time media transport; never trust client tenant IDs or client-provided authorization claims.

VISUAL WALKTHROUGH

See the system in motion

Establish tenant-scoped duplex media and reliable session control.

Process overviewAvailable even when interactive playback cannot load.
  1. Authenticate actor and tenant

  2. Negotiate media and control channels

  3. Pin codec and provider policy

  4. Create bounded session state

Loading interactive walkthrough…

Ingest and normalize microphone audio

The media gateway terminates the transport, validates session ownership and normalizes codec/sample-rate/channel metadata. Apply jitter buffering, packet-loss concealment and resampling only as required; preserve timing metadata and avoid unnecessary transcoding. Voice activity detection can operate on short windows to identify speech onset and silence candidates.

Use echo cancellation and noise suppression at the client/device where available, while retaining server-side quality checks. Bound per-session audio queues and apply backpressure or drop stale frames rather than allowing latency to grow without limit. Treat audio payloads as sensitive data with explicit retention and encryption controls.

Run streaming ASR with provisional and final text

Streaming ASR consumes overlapping audio windows and emits hypotheses with token/word timestamps, confidence and revision identifiers. A hypothesis is provisional until the recognizer marks a stable segment or the turn manager commits it. Preserve language, speaker and timing metadata where supported.

Do not trigger irreversible tools from a partial hypothesis. You may speculatively retrieve context or prepare a low-risk response, but bind speculative work to the hypothesis revision and discard it if later audio changes the intent. Final transcript events are immutable within the turn generation.

VISUAL WALKTHROUGH

See the system in motion

Follow provisional ASR hypotheses through stable transcript commitment.

Process overviewAvailable even when interactive playback cannot load.
  1. Normalize frames and timing

  2. Detect speech windows

  3. Emit provisional hypotheses

  4. Commit stable transcript segments

Loading interactive walkthrough…

Manage endpointing, pauses and turn ownership

Endpointing combines VAD silence, ASR stability, prosody and conversational policy. A fixed silence threshold is simple but can cut off slow speakers or add delay; adaptive endpointing uses language, pause patterns and interaction context while enforcing a maximum wait. Support explicit push-to-talk and client end-of-turn signals as overrides where appropriate.

Each user utterance receives a turnId and generation. On barge-in, increment the generation, stop or flush outgoing audio, cancel LLM/TTS work and invalidate late results. The turn manager is the authority for whether a result may be spoken.

VISUAL WALKTHROUGH

See the system in motion

See endpointing and explicit ownership of each conversational turn.

Process overviewAvailable even when interactive playback cannot load.
  1. Detect speech and silence

  2. Combine VAD and ASR stability

  3. Assign turn ID and generation

  4. Dispatch committed turn

Loading interactive walkthrough…

Orchestrate the assistant response

After a committed turn, assemble bounded conversation context, user preferences allowed by policy and the final transcript. Apply prompt version, model policy, deadline and token budget. Stream model output into a response buffer; distinguish text that is safe to speak from structured tool-call output.

A streaming response may be segmented into clauses for early TTS, but do not speak incomplete or policy-sensitive fragments prematurely. Track response generation and audio generation separately so a provider switch or interruption does not replay already delivered content.

Gate tool calls and high-impact actions

The model emits a typed tool proposal, not an executable command. Validate schema, committed user intent, tenant/actor authorization, resource scope, policy, rate limits and idempotency key. Read-only tools may run under narrower conditions; purchases, transfers, account changes or external messages require explicit confirmation and auditable consent.

Speech recognition uncertainty is a risk signal. For ambiguous names, amounts, dates or targets, ask a clarification question and repeat critical values before confirmation. Tool results return as structured data to the orchestrator; never let raw tool output override system policy or inject instructions into subsequent turns.

VISUAL WALKTHROUGH

See the system in motion

Gate spoken tool requests through schema, authorization and confirmation.

Process overviewAvailable even when interactive playback cannot load.
  1. Wait for committed transcript

  2. Validate typed tool arguments

  3. Authorize tenant and resource scope

  4. Confirm high-impact action then execute

Loading interactive walkthrough…

Synthesize and stream speech

TTS accepts validated text chunks and returns aligned audio chunks with sequence numbers, codec metadata and optional phoneme/word alignment. Begin playback when the first safe chunk is ready, but bound the playback buffer in milliseconds and keep a mapping from text spans to audio chunks for interruption and diagnostics.

Voice selection, language, prosody and pronunciation dictionaries are versioned configuration. If TTS fails after some audio has played, continue from a safe text boundary with a compatible provider or communicate a concise fallback; never restart the full answer blindly.

VISUAL WALKTHROUGH

See the system in motion

Stream aligned speech chunks through a bounded interruptible playback buffer.

Process overviewAvailable even when interactive playback cannot load.
  1. Emit safe text chunks

  2. Synthesize aligned audio chunks

  3. Apply sequence and generation IDs

  4. Play with bounded buffer and cancellation

Loading interactive walkthrough…

Implement interruption and cancellation correctly

Barge-in is a first-class state transition: detect incoming speech while assistant audio is playing, decide whether it is genuine user speech or echo, advance the turn generation, send an immediate stop signal to playback, cancel synthesis and model generation, and drain or discard queued chunks from older generations.

Cancellation is not proof that a provider stopped billing or processing. Record cancellation intent, provider acknowledgements and usage evidence separately. Late responses are rejected by generation fencing. Keep a short configurable acoustic echo guard to avoid the assistant interrupting itself.

VISUAL WALKTHROUGH

See the system in motion

Interrupt playback and reject stale model or TTS output using turn generations.

Process overviewAvailable even when interactive playback cannot load.
  1. Detect genuine user interruption

  2. Advance generation counter

  3. Stop playback and cancel work

  4. Reject stale audio chunks

Loading interactive walkthrough…

Store conversation state without coupling it to audio buffers

Persist durable session and turn metadata separately from ephemeral media buffers. Store committed transcript segments, assistant text, tool outcomes, prompt/model versions, consent and retention metadata. Audio recording is optional and policy-dependent; a transcript is also sensitive and should have independent access and retention rules.

Use bounded context windows with summarization or retrieval for long sessions. Summaries must preserve provenance and uncertainty; do not turn unverified ASR guesses into durable user facts. Session reconnect uses a cursor/sequence protocol and must not replay stale audio to the user.

Handle provider failure and reconnect

ASR, LLM and TTS are independent failure domains. Apply stage-specific deadlines, circuit breakers and provider compatibility checks. ASR interruption may require a user retry; LLM fallback must preserve the committed turn and prompt policy; TTS fallback should continue from an unspoken text boundary if alignment permits.

On network reconnect, authenticate the session again, verify expiry and generation, resume control state from a sequence cursor and renegotiate media. Do not assume buffered client audio can be replayed exactly once. If state is uncertain, ask the user to repeat rather than risk duplicate tool actions.

Protect voice, transcripts and tool authority

Authenticate users and authorize every session, transcript, tool and artifact by tenant and object. Encrypt media in transit and at rest, scope provider credentials, minimize raw-audio retention and honor consent, regional processing, deletion and legal-hold policy. Redact sensitive fields from traces and analytics.

Audio can contain secrets, personal data, adversarial instructions or background speech from bystanders. Treat recognized text as untrusted input. The voice channel does not weaken prompt-injection defenses or tool authorization; the same policy gate applies as in text interfaces.

Estimate concurrency, bandwidth and cost

Concurrent sessions ≈ peak active voice minutes per second × session duration distribution, adjusted for peak factor. Ingress bandwidth is concurrent sessions × codec bitrate; egress is separate. ASR/TTS cost is often duration-based while LLM cost is token-based, so attribute each stage to session and turn IDs.

Reserve capacity independently for media gateways, ASR streams, LLM tokens and TTS synthesis. Apply tenant quotas, fair scheduling and admission control before accepting sessions. When overloaded, prefer explicit wait/voice fallback or reduced optional features over silently increasing audio buffering and destroying conversational latency.

Measure voice quality and user outcomes

Trace session and turn state transitions with privacy-safe IDs. Measure ASR word error rate by language/accent/noise slice, endpointing false cuts and delays, interruption latency, TTS first-audio and intelligibility, tool success, task completion, turn abandonment and cost per successful task.

Offline replay cannot fully reproduce real conversational dynamics. Combine curated audio evaluation, human listening, network impairment tests, red-team tool scenarios and online product metrics. Avoid storing raw audio in general-purpose traces; use consented, access-controlled evaluation corpora.

Walk through a barge-in and provider failure

The assistant is speaking a response when the user interrupts with a correction. The gateway detects new speech; the turn manager advances generation 12 to 13, stops playback, cancels TTS and LLM work, and rejects any late generation-12 audio. Streaming ASR commits the correction, and the assistant responds to the new turn.

If the ASR provider times out mid-turn, persist the last committed segment and mark the partial utterance incomplete. Retry only the missing safe window if supported; otherwise ask the user to repeat. Tool execution is never repeated from speculative text, and all side effects remain protected by idempotency and confirmation.

Complete real-time voice assistant architecture

The full path is authenticated session → WebRTC/SIP media gateway → audio normalization/VAD → streaming ASR → turn manager/endpointing → context and policy orchestration → LLM/tool proposal → deterministic tool authorization/confirmation → streaming TTS → bounded playback buffer → user. Control events and media frames use separate channels; observability and privacy policy span every stage.

The final architecture must show barge-in as a first-class control path, generation fencing for late output, stage-specific provider failures, reconnect state and separate media/turn persistence. A voice assistant is not simply a text LLM with speech recognition and synthesis attached.

Complete animated architecture walkthrough

Follow the duplex media, streaming transcription, turn ownership, safe orchestration, speech synthesis and barge-in recovery path end to end.

VISUAL WALKTHROUGH

See the system in motion

Follow the full voice assistant lifecycle, including committed speech, generation-fenced output, tool policy and interruption.

Process overviewAvailable even when interactive playback cannot load.
  1. Authenticate and establish duplex session

  2. Stream normalized microphone audio

  3. Emit provisional and committed transcript

  4. Endpoint and assign turn generation

  5. Orchestrate response and authorize tool proposals

  6. Stream safe response text to TTS

  7. Return generation-fenced audio chunks

  8. Stop and fence stale output on barge-in

Loading interactive walkthrough…

Diagrams and relationships

Duplex voice session and control plane

Real-time media flows separately from reliable session and turn control.

Scroll horizontally on smaller screens, or enlarge the diagram.

Duplex voice session and control planeReal-time media flows separately from reliable session and turn control. Voice client to Media gateway: audio frames. Media gateway to Streaming ASR: normalized audio. Streaming ASR to Turn manager: partial/final text. Turn manager to LLM + policy: committed turn. LLM + policy to Streaming TTS: safe text chunks. Streaming TTS to Voice client: audio playback.audio framesnormalized audiopartial/final textcommitted turnsafe text chunksaudio playbackVoice clientMedia gatewayStreaming ASRTurn managerLLM + policyStreaming TTS
Duplex voice session and control planeReal-time media flows separately from reliable session and turn control. Voice client to Media gateway: audio frames. Media gateway to Streaming ASR: normalized audio. Streaming ASR to Turn manager: partial/final text. Turn manager to LLM + policy: committed turn. LLM + policy to Streaming TTS: safe text chunks. Streaming TTS to Voice client: audio playback.audio framesnormalized audiopartial/final textcommitted turnsafe text chunksaudio playbackVoice clientMedia gatewayStreaming ASRTurn managerLLM + policyStreaming TTS
Turn state and barge-in fencing

Every downstream result is tied to a turn generation; interruption invalidates older output.

Scroll horizontally on smaller screens, or enlarge the diagram.

Turn state and barge-in fencingEvery downstream result is tied to a turn generation; interruption invalidates older output. User speech to Barge-in detect: new audio. Barge-in detect to Turn generation: advance turn. Turn generation to Cancel LLM/TTS: cancel intent. Turn generation to Stop playback: stop signal. Cancel LLM/TTS to Reject stale chunks: late output. Stop playback to Reject stale chunks: old generation.new audioadvance turncancel intentstop signallate outputold generationUser speechBarge-in detectTurn generationCancel LLM/TTSStop playbackReject stale chunks
Turn state and barge-in fencingEvery downstream result is tied to a turn generation; interruption invalidates older output. User speech to Barge-in detect: new audio. Barge-in detect to Turn generation: advance turn. Turn generation to Cancel LLM/TTS: cancel intent. Turn generation to Stop playback: stop signal. Cancel LLM/TTS to Reject stale chunks: late output. Stop playback to Reject stale chunks: old generation.new audioadvance turncancel intentstop signallate outputold generationUser speechBarge-in detectTurn generationCancel LLM/TTSStop playbackReject stale chunks
Safe voice tool execution

Speech intent is committed, authorized and confirmed before side effects.

Scroll horizontally on smaller screens, or enlarge the diagram.

Safe voice tool executionSpeech intent is committed, authorized and confirmed before side effects. Committed transcript to Tool proposal: intent. Tool proposal to Policy gate: typed args. Policy gate to User confirmation: high impact. User confirmation to Scoped tool: approved action.intenttyped argshigh impactapproved actionCommitted transcriptTool proposalPolicy gateUser confirmationScoped tool
Safe voice tool executionSpeech intent is committed, authorized and confirmed before side effects. Committed transcript to Tool proposal: intent. Tool proposal to Policy gate: typed args. Policy gate to User confirmation: high impact. User confirmation to Scoped tool: approved action.intenttyped argshigh impactapproved actionCommitted transcriptTool proposalPolicy gateUser confirmationScoped tool

Engineer, Senior and Staff follow-ups

These are teaching lenses. Each answer changes the scope of reasoning; they are not official employer leveling criteria.

Engineer: Why separate partial and final ASR events?

Partial hypotheses can be revised. Irreversible actions must use committed intent; speculative retrieval or drafting can be tied to a revision and discarded if it changes.

Engineer: Why use WebRTC instead of WebSocket?

WebRTC provides media-oriented transport features such as jitter handling and congestion adaptation. WebSocket can work for controlled frame streaming, but the application must build pacing, loss and backpressure behavior itself.

Senior: How do you make barge-in reliable?

Advance a turn generation, stop playback immediately, cancel model and synthesis work, and reject every late chunk whose generation is stale. Keep playback buffers bounded and account for acoustic echo.

Senior: How would you reduce time-to-first-audio?

Measure each stage, overlap safe low-risk work, use streaming ASR/LLM/TTS, tune endpointing, keep providers warm and bound buffers. Do not trade away tool safety or speak unstable transcript content.

Senior: How do you handle uncertain recognition of an amount or account?

Treat ASR confidence and semantic ambiguity as risk signals. Ask a clarification, read back critical values and require explicit confirmation before a high-impact action.

Staff: How would you support multiple ASR and TTS providers?

Define normalized streaming contracts, capability and language metadata, deadlines, cancellation semantics and usage reporting. Route per stage and preserve committed turn state across compatible fallback.

Staff: How do you prevent stale speech after interruption?

Use monotonic turn generations and include generation/sequence in every text and audio chunk. The playback gateway validates before enqueue and again before rendering.

Staff: How do you evaluate conversational quality?

Combine ASR slices, endpointing and interruption metrics, human listening, task completion, tool correctness, safety and user outcomes. Replay is useful but does not replace live network and interaction testing.

Practice and self-review

Your practice draft

Design a real-time voice assistant for mobile and browser clients. Support duplex audio, streaming ASR, natural turn-taking, barge-in, safe tool calls and streaming TTS. Explain latency budgets, session state, provider failure, privacy and capacity.

Saved locally on this device. The worked answer does not score your draft. Existing lesson notes and completion are separate.

Progressive hints

Hint 1

Start with media session and separate control/media channels.

Hint 2

Define stage-level time-to-first-audio and end-of-turn SLOs.

Hint 3

Keep ASR partial hypotheses distinct from committed transcript.

Hint 4

Use turn IDs/generations to fence stale LLM and TTS output.

Hint 5

Require deterministic policy and confirmation for side effects.

Hint 6

Bound audio queues and make cancellation/reconnect explicit.

Reveal worked answer guidance

Authenticate a tenant-scoped session and negotiate WebRTC media plus a reliable control channel. Normalize audio and run VAD/streaming ASR, emitting provisional hypotheses and immutable committed segments. The turn manager performs endpointing and assigns turn generations. For a committed turn, assemble bounded context and invoke the LLM under deadline and policy. Validate typed tool proposals through deterministic authorization, idempotency and confirmation. Stream safe text chunks to TTS and bounded playback buffers. On barge-in, advance generation, stop playback, cancel LLM/TTS and reject stale chunks. Persist turn metadata separately from optional audio recordings. Apply stage-specific provider fallback, reconnect cursors, privacy retention, fair admission and per-stage observability.

Evidence to check in your answer

  • Media and control channels are separated and session identity is server-derived.
  • Latency is budgeted end-to-end, including endpointing and playback.
  • ASR provisional and committed transcript states are distinct.
  • Turn ownership and generation fencing prevent stale audio.
  • Barge-in cancels downstream work and stops playback promptly.
  • Tool calls are typed, authorized, idempotent and confirmed when high impact.
  • TTS streaming and playback buffers are bounded.
  • Provider fallback is stage-specific and does not duplicate side effects.
  • Audio, transcript, consent and retention policies are explicit.
  • Evaluation includes ASR slices, endpointing, interruption, intelligibility and task success.

For each item, record Missing, Prompted, Independent or Adaptive evidence. Adaptive means you can revise the reasoning under a changed constraint. Reading completion and readiness are separate.

Sources and verification scope

Original synthetic examples and reference calculations. Sources support narrowly scoped background; they do not establish production performance or independent chapter review.

Primary background links checked during authoring on 2026-10-19. Arithmetic, temporal eligibility, maturity and split reference functions are checked by the bounded local fixture suite. SQL is explanatory, not a migrated production query. No GPU, external dataset, model API or training job is required.