Learning pathsPricingChecking sign-in…
← ML curriculum

AIF-007 · FOUNDATION

Streaming, cancellation and backpressure

Reliable streaming generation requires an explicit event protocol, bounded buffering, reconnect semantics and cancellation fencing. A transport disconnect is not proof of completion, and a stop request is not proof that provider work or billing has ended.

Author self-review; independent technical review pending. Deterministic conceptual fixture; no live provider calls, production credentials or billing.

Prerequisites and scoped previews

Context windows and context engineering

Construct bounded, relevant and permission-safe context from instructions, conversation state, retrieved evidence and tool results.

Prompt contracts, structured outputs and tool calling

Separate trusted instructions from untrusted data, validate structured outputs and keep tool authority in application code.

What, when, why and how

Streaming is a protocol, not a sequence of text fragments

what
A generation stream is an ordered sequence of typed events such as accepted, delta, usage, completed, failed and cancelled.
when
Use streaming when users benefit from early partial output or long-running model work needs progress visibility.
why
Without event identity, sequence and terminal semantics, reconnects can duplicate text, stale workers can append after cancellation, and clients cannot distinguish completion from transport closure.
how
Persist run ID, generation ID, monotonically increasing sequence, event type and terminal state; expose resumable reads from committed events.

Cancellation must fence work and settle usage

what
Cancellation records intent, signals the worker and prevents late output from an obsolete generation from becoming visible.
when
Use it for user stop actions, superseded turns, deadline expiry and disconnect policies.
why
A browser abort does not prove the provider stopped computation or that billable tokens are zero.
how
Fence by generation ID, stop forwarding output, request provider cancellation, collect terminal usage evidence and reconcile reservation exactly once.

Backpressure protects both user experience and capacity

what
Backpressure bounds queued events and buffered bytes when the consumer is slower than generation.
when
Use it for slow clients, mobile networks, fan-out and high-concurrency streaming endpoints.
why
Unbounded buffers turn slow consumers into memory growth and increase tail latency for unrelated tenants.
how
Bound per-run and per-connection buffers, apply flow control, heartbeat idle connections and define when to detach and resume from the durable event log.

1. Define the stream event contract

A useful contract separates transport frames from application events. Each event carries runId, generationId, sequence, type and a typed payload. Common types are accepted, delta, usage, completed, failed and cancelled. A client renders deltas only when the generation matches the currently active turn and sequence is newer than the last committed event.

Persist the terminal event before acknowledging completion. A clean TCP close without a terminal application event is ambiguous: the client should reconnect or query run status, not assume success. Never infer success from a missing error frame.

VISUAL WALKTHROUGH

See the system in motion

Replay ordered committed generation events after reconnect without starting duplicate model work.

Process overviewAvailable even when interactive playback cannot load.
  1. Create a durable run and generation ID

  2. Commit ordered events with sequence numbers

  3. Deliver deltas to the active client

  4. Reconnect from the last committed sequence

  5. Deduplicate replayed events

  6. Return explicit terminal state

Loading interactive walkthrough…

2. Make reconnect and replay deterministic

The server commits events to a per-run ordered log before publishing them. The client reconnects with the last observed sequence and receives later committed events. Duplicate delivery is expected; the client deduplicates by runId + generationId + sequence. If retention has expired, return an explicit replay-window-expired response and a snapshot of current run state.

Do not start a second model generation merely because the browser reconnects. A retry of the HTTP stream attaches to the existing run. Starting new generation work requires a separate idempotency key and an explicit state transition.

3. Treat cancellation as a distributed state transition

Persist cancellation intent and increment or revoke the generation fence before asking the provider to stop. Every worker write checks that its generation is still active. This prevents late chunks from an uncooperative provider or delayed queue message from appearing after a user has moved on.

Track cancellation requested, provider stop acknowledged, worker terminal and usage reconciled as separate facts. If terminal usage never arrives, retain a bounded pending settlement and reconcile asynchronously; do not release all reserved budget merely because the browser closed.

VISUAL WALKTHROUGH

See the system in motion

Fence stale output before provider cancellation and settle usage only from terminal evidence.

Process overviewAvailable even when interactive playback cannot load.
  1. Persist cancellation intent

  2. Advance generation fence

  3. Signal provider and worker

  4. Reject late output from stale generation

  5. Collect terminal usage evidence

  6. Settle reservation exactly once

Loading interactive walkthrough…

4. Bound buffers and deadlines

Set limits for buffered bytes, queued events, idle time, total generation time and maximum output tokens. When a consumer falls behind, pause upstream reads where supported, detach the client and allow replay, or terminate with an explicit resumable status. Avoid holding an application worker open indefinitely for a disconnected browser.

Propagate one absolute deadline through admission, queue wait, provider call and stream delivery. Per-stage timeouts must fit inside the remaining deadline; retries must not reset the user's total latency budget.

Diagrams and relationships

Durable stream and cancellation fence

The event log is the source of truth; cancellation fences stale generations before provider stop and usage settlement.

Scroll horizontally on smaller screens, or enlarge the diagram.

Durable stream and cancellation fenceThe event log is the source of truth; cancellation fences stale generations before provider stop and usage settlement. Request + run ID to Provider worker: admit. Provider worker to Ordered event log: commit events. Ordered event log to Client replay: replay by seq. Cancel + fence to Provider worker: stop + reject stale. Provider worker to Usage settlement: terminal usage.admitcommit eventsreplay by seqstop + reject staleterminal usageRequest + run IDProvider workerOrdered event logClient replayCancel + fenceUsage settlement
Durable stream and cancellation fenceThe event log is the source of truth; cancellation fences stale generations before provider stop and usage settlement. Request + run ID to Provider worker: admit. Provider worker to Ordered event log: commit events. Ordered event log to Client replay: replay by seq. Cancel + fence to Provider worker: stop + reject stale. Provider worker to Usage settlement: terminal usage.admitcommit eventsreplay by seqstop + reject staleterminal usageRequest + run IDProvider workerOrdered event logClient replayCancel + fenceUsage settlement

Engineer, Senior and Staff follow-ups

These are teaching lenses. Each answer changes the scope of reasoning; they are not official employer leveling criteria.

Engineer: Why should a reconnect not create a new model request?

A reconnect is a transport recovery operation, not a new user intent. Attach to the existing run and replay committed events using the last sequence; a new generation needs a new idempotency key and explicit request.

Engineer: What does a generation fence prevent?

It prevents late chunks or delayed worker writes from an old generation from being committed after cancellation, retry or a newer user turn has superseded it.

Senior: How do you handle a provider that ignores cancellation?

Stop forwarding and committing output after the fence, keep the run in a cancellation-pending state, reconcile terminal usage asynchronously, and cap outstanding reservations and worker lifetime.

Senior: How would you design SSE replay?

Persist ordered run events, expose sequence IDs, let clients reconnect with the last committed ID, deduplicate repeated events and return explicit snapshot/retention semantics when the replay window is unavailable.

Staff: How do you prevent slow clients from consuming unbounded memory?

Bound per-run and per-connection buffers, enforce idle and total deadlines, use upstream flow control when possible, otherwise detach and replay from durable storage, and monitor lag and disconnect rates by tenant.

Staff: What is the correct billing behavior after cancellation?

Cancellation intent is not terminal usage. Keep a bounded reservation until provider or worker terminal evidence arrives, reconcile late/duplicate reports idempotently and release only the unused remainder.

Practice and self-review

Your practice draft

Design streaming chat for 100,000 concurrent sessions. Clients can reconnect, users can stop generation, providers may ignore cancellation briefly, and token usage arrives after the final text chunk.

Saved locally on this device. The worked answer does not score your draft. Existing lesson notes and completion are separate.

Progressive hints

Hint 1

Define event identity, ordering, terminal states and replay retention.

Hint 2

Fence stale generations before signaling provider cancellation; keep transport close separate from run completion.

Hint 3

Reconcile usage exactly once and bound slow-consumer buffers and deadlines.

Reveal worked answer guidance

Use a durable run record and ordered event log keyed by tenant/run/generation. Workers append sequenced events conditionally while their generation fence is active. SSE/WebSocket gateways read the committed log and resume from Last-Event-ID. Cancellation persists intent and revokes the generation before signaling the provider; late chunks are rejected. A reconciliation worker waits for terminal usage evidence, deduplicates provider reports and settles a pre-reserved budget once. Gateways bound buffers and detach slow clients with replay support.

Evidence to check in your answer

  • Reconnect attaches to an existing run and deduplicates events.
  • Cancellation fences stale writes and distinguishes provider stop from user intent.
  • Usage settlement is idempotent and based on terminal evidence.
  • Buffers, deadlines and concurrency are explicitly bounded.

For each item, record Missing, Prompted, Independent or Adaptive evidence. Adaptive means you can revise the reasoning under a changed constraint. Reading completion and readiness are separate.

Sources and verification scope

Original synthetic examples and reference calculations. Sources support narrowly scoped background; they do not establish production performance or independent chapter review.

SRE workbook: data processing pipelines ↗ · Pipeline SLOs and operational ownership; do not copy source workload numbers as course production results.

vLLM automatic prefix caching ↗ · Prefix KV reuse and its prefill-specific benefit; does not establish tenant authorization or cross-tenant cache policy.

OpenTelemetry GenAI token metrics ↗ · Token usage counters versus per-operation histograms; use billable usage when both billable and consumed usage are available.

Primary background links checked during authoring on 2026-10-19. Arithmetic, temporal eligibility, maturity and split reference functions are checked by the bounded local fixture suite. SQL is explanatory, not a migrated production query. No GPU, external dataset, model API or training job is required.