Prerequisites and scoped previews
Context windows and context engineering
Construct bounded, relevant and permission-safe context from instructions, conversation state, retrieved evidence and tool results.
Prompt contracts, structured outputs and tool calling
Separate trusted instructions from untrusted data, validate structured outputs and keep tool authority in application code.
What, when, why and how
Streaming is a protocol, not a sequence of text fragments
- what
- A generation stream is an ordered sequence of typed events such as accepted, delta, usage, completed, failed and cancelled.
- when
- Use streaming when users benefit from early partial output or long-running model work needs progress visibility.
- why
- Without event identity, sequence and terminal semantics, reconnects can duplicate text, stale workers can append after cancellation, and clients cannot distinguish completion from transport closure.
- how
- Persist run ID, generation ID, monotonically increasing sequence, event type and terminal state; expose resumable reads from committed events.
Cancellation must fence work and settle usage
- what
- Cancellation records intent, signals the worker and prevents late output from an obsolete generation from becoming visible.
- when
- Use it for user stop actions, superseded turns, deadline expiry and disconnect policies.
- why
- A browser abort does not prove the provider stopped computation or that billable tokens are zero.
- how
- Fence by generation ID, stop forwarding output, request provider cancellation, collect terminal usage evidence and reconcile reservation exactly once.
Backpressure protects both user experience and capacity
- what
- Backpressure bounds queued events and buffered bytes when the consumer is slower than generation.
- when
- Use it for slow clients, mobile networks, fan-out and high-concurrency streaming endpoints.
- why
- Unbounded buffers turn slow consumers into memory growth and increase tail latency for unrelated tenants.
- how
- Bound per-run and per-connection buffers, apply flow control, heartbeat idle connections and define when to detach and resume from the durable event log.
1. Define the stream event contract
A useful contract separates transport frames from application events. Each event carries runId, generationId, sequence, type and a typed payload. Common types are accepted, delta, usage, completed, failed and cancelled. A client renders deltas only when the generation matches the currently active turn and sequence is newer than the last committed event.
Persist the terminal event before acknowledging completion. A clean TCP close without a terminal application event is ambiguous: the client should reconnect or query run status, not assume success. Never infer success from a missing error frame.
VISUAL WALKTHROUGH
See the system in motion
Replay ordered committed generation events after reconnect without starting duplicate model work.
Create a durable run and generation ID
Commit ordered events with sequence numbers
Deliver deltas to the active client
Reconnect from the last committed sequence
Deduplicate replayed events
Return explicit terminal state
Loading interactive walkthrough…
2. Make reconnect and replay deterministic
The server commits events to a per-run ordered log before publishing them. The client reconnects with the last observed sequence and receives later committed events. Duplicate delivery is expected; the client deduplicates by runId + generationId + sequence. If retention has expired, return an explicit replay-window-expired response and a snapshot of current run state.
Do not start a second model generation merely because the browser reconnects. A retry of the HTTP stream attaches to the existing run. Starting new generation work requires a separate idempotency key and an explicit state transition.
3. Treat cancellation as a distributed state transition
Persist cancellation intent and increment or revoke the generation fence before asking the provider to stop. Every worker write checks that its generation is still active. This prevents late chunks from an uncooperative provider or delayed queue message from appearing after a user has moved on.
Track cancellation requested, provider stop acknowledged, worker terminal and usage reconciled as separate facts. If terminal usage never arrives, retain a bounded pending settlement and reconcile asynchronously; do not release all reserved budget merely because the browser closed.
VISUAL WALKTHROUGH
See the system in motion
Fence stale output before provider cancellation and settle usage only from terminal evidence.
Persist cancellation intent
Advance generation fence
Signal provider and worker
Reject late output from stale generation
Collect terminal usage evidence
Settle reservation exactly once
Loading interactive walkthrough…
4. Bound buffers and deadlines
Set limits for buffered bytes, queued events, idle time, total generation time and maximum output tokens. When a consumer falls behind, pause upstream reads where supported, detach the client and allow replay, or terminate with an explicit resumable status. Avoid holding an application worker open indefinitely for a disconnected browser.
Propagate one absolute deadline through admission, queue wait, provider call and stream delivery. Per-stage timeouts must fit inside the remaining deadline; retries must not reset the user's total latency budget.
Diagrams and relationships
The event log is the source of truth; cancellation fences stale generations before provider stop and usage settlement.
Scroll horizontally on smaller screens, or enlarge the diagram.
Engineer, Senior and Staff follow-ups
These are teaching lenses. Each answer changes the scope of reasoning; they are not official employer leveling criteria.
Engineer: Why should a reconnect not create a new model request?
A reconnect is a transport recovery operation, not a new user intent. Attach to the existing run and replay committed events using the last sequence; a new generation needs a new idempotency key and explicit request.
Engineer: What does a generation fence prevent?
It prevents late chunks or delayed worker writes from an old generation from being committed after cancellation, retry or a newer user turn has superseded it.
Senior: How do you handle a provider that ignores cancellation?
Stop forwarding and committing output after the fence, keep the run in a cancellation-pending state, reconcile terminal usage asynchronously, and cap outstanding reservations and worker lifetime.
Senior: How would you design SSE replay?
Persist ordered run events, expose sequence IDs, let clients reconnect with the last committed ID, deduplicate repeated events and return explicit snapshot/retention semantics when the replay window is unavailable.
Staff: How do you prevent slow clients from consuming unbounded memory?
Bound per-run and per-connection buffers, enforce idle and total deadlines, use upstream flow control when possible, otherwise detach and replay from durable storage, and monitor lag and disconnect rates by tenant.
Staff: What is the correct billing behavior after cancellation?
Cancellation intent is not terminal usage. Keep a bounded reservation until provider or worker terminal evidence arrives, reconcile late/duplicate reports idempotently and release only the unused remainder.
Practice and self-review
Your practice draft
Design streaming chat for 100,000 concurrent sessions. Clients can reconnect, users can stop generation, providers may ignore cancellation briefly, and token usage arrives after the final text chunk.
Saved locally on this device. The worked answer does not score your draft. Existing lesson notes and completion are separate.
Progressive hints
Hint 1
Define event identity, ordering, terminal states and replay retention.
Hint 2
Fence stale generations before signaling provider cancellation; keep transport close separate from run completion.
Hint 3
Reconcile usage exactly once and bound slow-consumer buffers and deadlines.
Reveal worked answer guidance
Use a durable run record and ordered event log keyed by tenant/run/generation. Workers append sequenced events conditionally while their generation fence is active. SSE/WebSocket gateways read the committed log and resume from Last-Event-ID. Cancellation persists intent and revokes the generation before signaling the provider; late chunks are rejected. A reconciliation worker waits for terminal usage evidence, deduplicates provider reports and settles a pre-reserved budget once. Gateways bound buffers and detach slow clients with replay support.
Evidence to check in your answer
- Reconnect attaches to an existing run and deduplicates events.
- Cancellation fences stale writes and distinguishes provider stop from user intent.
- Usage settlement is idempotent and based on terminal evidence.
- Buffers, deadlines and concurrency are explicitly bounded.
For each item, record Missing, Prompted, Independent or Adaptive evidence. Adaptive means you can revise the reasoning under a changed constraint. Reading completion and readiness are separate.
Sources and verification scope
Original synthetic examples and reference calculations. Sources support narrowly scoped background; they do not establish production performance or independent chapter review.
SRE workbook: data processing pipelines ↗ · Pipeline SLOs and operational ownership; do not copy source workload numbers as course production results.
vLLM automatic prefix caching ↗ · Prefix KV reuse and its prefill-specific benefit; does not establish tenant authorization or cross-tenant cache policy.
OpenTelemetry GenAI token metrics ↗ · Token usage counters versus per-operation histograms; use billable usage when both billable and consumed usage are available.
Primary background links checked during authoring on 2026-10-19. Arithmetic, temporal eligibility, maturity and split reference functions are checked by the bounded local fixture suite. SQL is explanatory, not a migrated production query. No GPU, external dataset, model API or training job is required.