Prerequisites and scoped previews
Leakage, splits and evaluation contamination
Prevent future information, entity overlap, preprocessing leakage and test-set reuse.
Deep models and sequence context
Justify neural/sequence architectures from data/task structure and resource constraints.
Scoped prerequisite preview. The complete chapter is planned; this preview does not count as authored coverage.
Regression and forecasting evaluation
Match MAE/RMSE/quantile/interval metrics to horizon and error costs; backtest over time.
Scoped prerequisite preview. The complete chapter is planned; this preview does not count as authored coverage.
Multi-tenant LLM inference gateway
Tenant-safe model routing, atomic budget admission, KV-aware serving, resumable streams and evidence-based usage settlement.
Multimodal video understanding platform
Design temporal video/audio ingestion, multimodal retrieval and grounded timestamped answers.
What, when, why and how
Treat voice as a real-time interaction, not chat with a microphone
- what
- Voice has continuous media transport, partial hypotheses, turn boundaries, audio rendering and interruption.
- when
- For conversational assistants, contact centers and hands-free interfaces.
- why
- A request/response chat architecture adds latency and cannot handle overlapping speech or barge-in.
- how
- Model media sessions, turns and utterance segments separately; use streaming transport and explicit deadlines.
Budget latency across the whole turn
- what
- Perceived latency is the sum of capture, transport, endpointing, ASR, orchestration, model first-token, TTS first-audio and playback buffers.
- when
- When users expect natural conversation.
- why
- Optimizing only model inference leaves endpointing or audio buffering as the dominant delay.
- how
- Define stage budgets and measure p50/p95 time-to-first-audio and end-of-turn completion.
Separate partial hypotheses from committed transcript
- what
- Streaming ASR revises provisional text until a stable segment is committed.
- when
- For low-latency turn-taking and live captions.
- why
- Acting on unstable words can trigger wrong tools or responses.
- how
- Track hypothesis revisions, stability and finalization; permit low-risk speculative work but gate side effects on committed intent.
Make turn ownership explicit
- what
- A turn manager owns turn IDs, endpointing, interruption, cancellation and output state.
- when
- For duplex audio where user speech can overlap assistant speech.
- why
- Without explicit ownership, stale LLM/TTS chunks can play after the user interrupts.
- how
- Use monotonically increasing turn generations and fence every downstream result against the active turn.
Stream TTS with playback-aware backpressure
- what
- TTS can emit audio chunks before the complete response is ready.
- when
- When minimizing time to first audio.
- why
- Unbounded synthesis or playback buffers increase latency and make interruption sluggish.
- how
- Bound queued audio duration, propagate cancellation and keep text/audio chunk alignment metadata.
Keep tools behind deterministic policy gates
- what
- The model proposes structured actions; an authorization layer validates identity, arguments, risk and confirmation.
- when
- When voice can access accounts or perform actions.
- why
- Speech recognition errors and prompt injection can produce unintended side effects.
- how
- Require committed transcript, schema validation, least privilege, idempotency and human confirmation for high-impact actions.
Design graceful degradation across providers
- what
- ASR, LLM and TTS are separate failure domains with different fallback options.
- when
- For availability-sensitive voice sessions.
- why
- Retrying the entire turn can duplicate actions and discard already-produced audio.
- how
- Persist turn state, retry only safe stages, switch compatible providers and clearly signal degraded mode.
Frame the real-time voice assistant
A user speaks naturally, may pause or interrupt, and expects the assistant to respond in speech. The system must stream microphone audio, produce provisional and final transcripts, decide when a turn is complete, generate a response, optionally call tools and stream synthesized audio back while remaining interruptible.
Synthetic exercise assumptions: 100,000 daily active users, 8 minutes average voice time/day, 5 concurrent sessions per 1,000 DAU at peak, 16 kHz mono input and 24 kHz output. These are interview assumptions, not production measurements. Calculate concurrency from active session minutes and peak factor, not DAU alone.
Define interaction requirements and SLOs
Functional requirements: session establishment, streaming audio ingress/egress, partial/final ASR, endpointing, turn management, barge-in, conversational context, safe tools, streaming TTS, reconnect and session termination. Non-functional requirements: low time-to-first-audio, bounded end-of-turn latency, intelligibility, interruption responsiveness, availability, privacy and cost.
Track stage-level p50/p95/p99, time-to-first-transcript, endpointing delay, time-to-first-model-token, time-to-first-audio, total turn latency, barge-in-to-audio-stop, dropped audio, jitter-buffer underruns and task success. Set product-specific targets after user research; do not assert universal latency thresholds.
Choose the media transport and session contract
For browser/mobile interactive audio, WebRTC is usually a strong choice when low-latency duplex media, jitter handling and network adaptation matter. WebSocket can be appropriate for controlled environments, simpler audio-frame streaming or text/control channels, but the application must explicitly handle framing, pacing, loss and congestion. SIP/PSTN gateways are a separate ingress boundary for telephony.
A session record pins tenant, authenticated actor, consent state, codec/sample rates, locale, provider/model policy, expiry and session generation. Media credentials are short-lived and scoped. Separate the reliable control channel (session, turn, tool status) from real-time media transport; never trust client tenant IDs or client-provided authorization claims.
VISUAL WALKTHROUGH
See the system in motion
Establish tenant-scoped duplex media and reliable session control.
Authenticate actor and tenant
Negotiate media and control channels
Pin codec and provider policy
Create bounded session state
Loading interactive walkthrough…
Ingest and normalize microphone audio
The media gateway terminates the transport, validates session ownership and normalizes codec/sample-rate/channel metadata. Apply jitter buffering, packet-loss concealment and resampling only as required; preserve timing metadata and avoid unnecessary transcoding. Voice activity detection can operate on short windows to identify speech onset and silence candidates.
Use echo cancellation and noise suppression at the client/device where available, while retaining server-side quality checks. Bound per-session audio queues and apply backpressure or drop stale frames rather than allowing latency to grow without limit. Treat audio payloads as sensitive data with explicit retention and encryption controls.
Run streaming ASR with provisional and final text
Streaming ASR consumes overlapping audio windows and emits hypotheses with token/word timestamps, confidence and revision identifiers. A hypothesis is provisional until the recognizer marks a stable segment or the turn manager commits it. Preserve language, speaker and timing metadata where supported.
Do not trigger irreversible tools from a partial hypothesis. You may speculatively retrieve context or prepare a low-risk response, but bind speculative work to the hypothesis revision and discard it if later audio changes the intent. Final transcript events are immutable within the turn generation.
VISUAL WALKTHROUGH
See the system in motion
Follow provisional ASR hypotheses through stable transcript commitment.
Normalize frames and timing
Detect speech windows
Emit provisional hypotheses
Commit stable transcript segments
Loading interactive walkthrough…
Manage endpointing, pauses and turn ownership
Endpointing combines VAD silence, ASR stability, prosody and conversational policy. A fixed silence threshold is simple but can cut off slow speakers or add delay; adaptive endpointing uses language, pause patterns and interaction context while enforcing a maximum wait. Support explicit push-to-talk and client end-of-turn signals as overrides where appropriate.
Each user utterance receives a turnId and generation. On barge-in, increment the generation, stop or flush outgoing audio, cancel LLM/TTS work and invalidate late results. The turn manager is the authority for whether a result may be spoken.
VISUAL WALKTHROUGH
See the system in motion
See endpointing and explicit ownership of each conversational turn.
Detect speech and silence
Combine VAD and ASR stability
Assign turn ID and generation
Dispatch committed turn
Loading interactive walkthrough…
Orchestrate the assistant response
After a committed turn, assemble bounded conversation context, user preferences allowed by policy and the final transcript. Apply prompt version, model policy, deadline and token budget. Stream model output into a response buffer; distinguish text that is safe to speak from structured tool-call output.
A streaming response may be segmented into clauses for early TTS, but do not speak incomplete or policy-sensitive fragments prematurely. Track response generation and audio generation separately so a provider switch or interruption does not replay already delivered content.
Gate tool calls and high-impact actions
The model emits a typed tool proposal, not an executable command. Validate schema, committed user intent, tenant/actor authorization, resource scope, policy, rate limits and idempotency key. Read-only tools may run under narrower conditions; purchases, transfers, account changes or external messages require explicit confirmation and auditable consent.
Speech recognition uncertainty is a risk signal. For ambiguous names, amounts, dates or targets, ask a clarification question and repeat critical values before confirmation. Tool results return as structured data to the orchestrator; never let raw tool output override system policy or inject instructions into subsequent turns.
VISUAL WALKTHROUGH
See the system in motion
Gate spoken tool requests through schema, authorization and confirmation.
Wait for committed transcript
Validate typed tool arguments
Authorize tenant and resource scope
Confirm high-impact action then execute
Loading interactive walkthrough…
Synthesize and stream speech
TTS accepts validated text chunks and returns aligned audio chunks with sequence numbers, codec metadata and optional phoneme/word alignment. Begin playback when the first safe chunk is ready, but bound the playback buffer in milliseconds and keep a mapping from text spans to audio chunks for interruption and diagnostics.
Voice selection, language, prosody and pronunciation dictionaries are versioned configuration. If TTS fails after some audio has played, continue from a safe text boundary with a compatible provider or communicate a concise fallback; never restart the full answer blindly.
VISUAL WALKTHROUGH
See the system in motion
Stream aligned speech chunks through a bounded interruptible playback buffer.
Emit safe text chunks
Synthesize aligned audio chunks
Apply sequence and generation IDs
Play with bounded buffer and cancellation
Loading interactive walkthrough…
Implement interruption and cancellation correctly
Barge-in is a first-class state transition: detect incoming speech while assistant audio is playing, decide whether it is genuine user speech or echo, advance the turn generation, send an immediate stop signal to playback, cancel synthesis and model generation, and drain or discard queued chunks from older generations.
Cancellation is not proof that a provider stopped billing or processing. Record cancellation intent, provider acknowledgements and usage evidence separately. Late responses are rejected by generation fencing. Keep a short configurable acoustic echo guard to avoid the assistant interrupting itself.
VISUAL WALKTHROUGH
See the system in motion
Interrupt playback and reject stale model or TTS output using turn generations.
Detect genuine user interruption
Advance generation counter
Stop playback and cancel work
Reject stale audio chunks
Loading interactive walkthrough…
Store conversation state without coupling it to audio buffers
Persist durable session and turn metadata separately from ephemeral media buffers. Store committed transcript segments, assistant text, tool outcomes, prompt/model versions, consent and retention metadata. Audio recording is optional and policy-dependent; a transcript is also sensitive and should have independent access and retention rules.
Use bounded context windows with summarization or retrieval for long sessions. Summaries must preserve provenance and uncertainty; do not turn unverified ASR guesses into durable user facts. Session reconnect uses a cursor/sequence protocol and must not replay stale audio to the user.
Handle provider failure and reconnect
ASR, LLM and TTS are independent failure domains. Apply stage-specific deadlines, circuit breakers and provider compatibility checks. ASR interruption may require a user retry; LLM fallback must preserve the committed turn and prompt policy; TTS fallback should continue from an unspoken text boundary if alignment permits.
On network reconnect, authenticate the session again, verify expiry and generation, resume control state from a sequence cursor and renegotiate media. Do not assume buffered client audio can be replayed exactly once. If state is uncertain, ask the user to repeat rather than risk duplicate tool actions.
Protect voice, transcripts and tool authority
Authenticate users and authorize every session, transcript, tool and artifact by tenant and object. Encrypt media in transit and at rest, scope provider credentials, minimize raw-audio retention and honor consent, regional processing, deletion and legal-hold policy. Redact sensitive fields from traces and analytics.
Audio can contain secrets, personal data, adversarial instructions or background speech from bystanders. Treat recognized text as untrusted input. The voice channel does not weaken prompt-injection defenses or tool authorization; the same policy gate applies as in text interfaces.
Estimate concurrency, bandwidth and cost
Concurrent sessions ≈ peak active voice minutes per second × session duration distribution, adjusted for peak factor. Ingress bandwidth is concurrent sessions × codec bitrate; egress is separate. ASR/TTS cost is often duration-based while LLM cost is token-based, so attribute each stage to session and turn IDs.
Reserve capacity independently for media gateways, ASR streams, LLM tokens and TTS synthesis. Apply tenant quotas, fair scheduling and admission control before accepting sessions. When overloaded, prefer explicit wait/voice fallback or reduced optional features over silently increasing audio buffering and destroying conversational latency.
Measure voice quality and user outcomes
Trace session and turn state transitions with privacy-safe IDs. Measure ASR word error rate by language/accent/noise slice, endpointing false cuts and delays, interruption latency, TTS first-audio and intelligibility, tool success, task completion, turn abandonment and cost per successful task.
Offline replay cannot fully reproduce real conversational dynamics. Combine curated audio evaluation, human listening, network impairment tests, red-team tool scenarios and online product metrics. Avoid storing raw audio in general-purpose traces; use consented, access-controlled evaluation corpora.
Walk through a barge-in and provider failure
The assistant is speaking a response when the user interrupts with a correction. The gateway detects new speech; the turn manager advances generation 12 to 13, stops playback, cancels TTS and LLM work, and rejects any late generation-12 audio. Streaming ASR commits the correction, and the assistant responds to the new turn.
If the ASR provider times out mid-turn, persist the last committed segment and mark the partial utterance incomplete. Retry only the missing safe window if supported; otherwise ask the user to repeat. Tool execution is never repeated from speculative text, and all side effects remain protected by idempotency and confirmation.
Complete real-time voice assistant architecture
The full path is authenticated session → WebRTC/SIP media gateway → audio normalization/VAD → streaming ASR → turn manager/endpointing → context and policy orchestration → LLM/tool proposal → deterministic tool authorization/confirmation → streaming TTS → bounded playback buffer → user. Control events and media frames use separate channels; observability and privacy policy span every stage.
The final architecture must show barge-in as a first-class control path, generation fencing for late output, stage-specific provider failures, reconnect state and separate media/turn persistence. A voice assistant is not simply a text LLM with speech recognition and synthesis attached.
Complete animated architecture walkthrough
Follow the duplex media, streaming transcription, turn ownership, safe orchestration, speech synthesis and barge-in recovery path end to end.
VISUAL WALKTHROUGH
See the system in motion
Follow the full voice assistant lifecycle, including committed speech, generation-fenced output, tool policy and interruption.
Authenticate and establish duplex session
Stream normalized microphone audio
Emit provisional and committed transcript
Endpoint and assign turn generation
Orchestrate response and authorize tool proposals
Stream safe response text to TTS
Return generation-fenced audio chunks
Stop and fence stale output on barge-in
Loading interactive walkthrough…
Diagrams and relationships
Real-time media flows separately from reliable session and turn control.
Scroll horizontally on smaller screens, or enlarge the diagram.
Every downstream result is tied to a turn generation; interruption invalidates older output.
Scroll horizontally on smaller screens, or enlarge the diagram.
Speech intent is committed, authorized and confirmed before side effects.
Scroll horizontally on smaller screens, or enlarge the diagram.
Engineer, Senior and Staff follow-ups
These are teaching lenses. Each answer changes the scope of reasoning; they are not official employer leveling criteria.
Engineer: Why separate partial and final ASR events?
Partial hypotheses can be revised. Irreversible actions must use committed intent; speculative retrieval or drafting can be tied to a revision and discarded if it changes.
Engineer: Why use WebRTC instead of WebSocket?
WebRTC provides media-oriented transport features such as jitter handling and congestion adaptation. WebSocket can work for controlled frame streaming, but the application must build pacing, loss and backpressure behavior itself.
Senior: How do you make barge-in reliable?
Advance a turn generation, stop playback immediately, cancel model and synthesis work, and reject every late chunk whose generation is stale. Keep playback buffers bounded and account for acoustic echo.
Senior: How would you reduce time-to-first-audio?
Measure each stage, overlap safe low-risk work, use streaming ASR/LLM/TTS, tune endpointing, keep providers warm and bound buffers. Do not trade away tool safety or speak unstable transcript content.
Senior: How do you handle uncertain recognition of an amount or account?
Treat ASR confidence and semantic ambiguity as risk signals. Ask a clarification, read back critical values and require explicit confirmation before a high-impact action.
Staff: How would you support multiple ASR and TTS providers?
Define normalized streaming contracts, capability and language metadata, deadlines, cancellation semantics and usage reporting. Route per stage and preserve committed turn state across compatible fallback.
Staff: How do you prevent stale speech after interruption?
Use monotonic turn generations and include generation/sequence in every text and audio chunk. The playback gateway validates before enqueue and again before rendering.
Staff: How do you evaluate conversational quality?
Combine ASR slices, endpointing and interruption metrics, human listening, task completion, tool correctness, safety and user outcomes. Replay is useful but does not replace live network and interaction testing.
Practice and self-review
Your practice draft
Design a real-time voice assistant for mobile and browser clients. Support duplex audio, streaming ASR, natural turn-taking, barge-in, safe tool calls and streaming TTS. Explain latency budgets, session state, provider failure, privacy and capacity.
Saved locally on this device. The worked answer does not score your draft. Existing lesson notes and completion are separate.
Progressive hints
Hint 1
Start with media session and separate control/media channels.
Hint 2
Define stage-level time-to-first-audio and end-of-turn SLOs.
Hint 3
Keep ASR partial hypotheses distinct from committed transcript.
Hint 4
Use turn IDs/generations to fence stale LLM and TTS output.
Hint 5
Require deterministic policy and confirmation for side effects.
Hint 6
Bound audio queues and make cancellation/reconnect explicit.
Reveal worked answer guidance
Authenticate a tenant-scoped session and negotiate WebRTC media plus a reliable control channel. Normalize audio and run VAD/streaming ASR, emitting provisional hypotheses and immutable committed segments. The turn manager performs endpointing and assigns turn generations. For a committed turn, assemble bounded context and invoke the LLM under deadline and policy. Validate typed tool proposals through deterministic authorization, idempotency and confirmation. Stream safe text chunks to TTS and bounded playback buffers. On barge-in, advance generation, stop playback, cancel LLM/TTS and reject stale chunks. Persist turn metadata separately from optional audio recordings. Apply stage-specific provider fallback, reconnect cursors, privacy retention, fair admission and per-stage observability.
Evidence to check in your answer
- Media and control channels are separated and session identity is server-derived.
- Latency is budgeted end-to-end, including endpointing and playback.
- ASR provisional and committed transcript states are distinct.
- Turn ownership and generation fencing prevent stale audio.
- Barge-in cancels downstream work and stops playback promptly.
- Tool calls are typed, authorized, idempotent and confirmed when high impact.
- TTS streaming and playback buffers are bounded.
- Provider fallback is stage-specific and does not duplicate side effects.
- Audio, transcript, consent and retention policies are explicit.
- Evaluation includes ASR slices, endpointing, interruption, intelligibility and task success.
For each item, record Missing, Prompted, Independent or Adaptive evidence. Adaptive means you can revise the reasoning under a changed constraint. Reading completion and readiness are separate.
Sources and verification scope
Original synthetic examples and reference calculations. Sources support narrowly scoped background; they do not establish production performance or independent chapter review.
Primary background links checked during authoring on 2026-10-19. Arithmetic, temporal eligibility, maturity and split reference functions are checked by the bounded local fixture suite. SQL is explanatory, not a migrated production query. No GPU, external dataset, model API or training job is required.