Schema, API and Event Evolution

Schema, API and Event Evolution

A rolling deployment changes a live conversation among versions. Storage rows, HTTP clients, event producers, consumers, connectors, workflow histories and repair tools may upgrade at different times. A syntactically valid new message can still change business meaning or break a retained replay. Define the compatibility horizon and the exact writer/reader pairs before choosing a deployment order.

Start with one contract and one change

This original exercise emits a tenant-scoped order event with event_id, order_id, currency and integer amount_minor. Version 2 adds an optional display_label; its absence means no label. It does not change the amount's units, required fields, event identity or ordering scope. A consumer ignoring this field can still process the original business action. A newly required field without a supported old-message default is a different, potentially breaking change.

These names describe the chosen serialization/registry contract. Read the specific format's rules. A JSON client that rejects unknown properties differs from one that ignores them; generated code may fail on an enum it never handled; a field can be wire-readable while its units or authorization meaning changed. A registry can enforce selected schema rules, but not prove application semantics, privacy, affordability or safe compensation.

Work the reader/writer matrix

For the exercise, R1 requires the original four payload fields, validates their types and ignores unrecognized properties. R2 accepts the same required fields, reads display_label if present, and handles absence explicitly. Both validate tenant and operation ownership independently of schema parsing.

The last row requires producer intent, domain validation and representative semantic fixtures, not just a schema registry. Preserve integer currency precision and documented units. A consumer must not treat an unknown payment state as success just to keep deserialization green.

{
  "event_id": "E7",
  "event_type": "order.accepted",
  "schema_version": 2,
  "tenant_id": "T1",
  "aggregate_id": "O7",
  "aggregate_version": 4,
  "payload": {"order_id": "O7", "currency": "USD", "amount_minor": 1250,
              "display_label": "Classroom supplies"}
}

The schema version identifies representation. Aggregate version orders changes for one entity under its source contract. Event ID names one logical event across redelivery. These fields are not interchangeable, and none grants tenant access.

Upgrade consumers before activating an allowed new producer

1. Inventory active clients, consumers, workflow code, archives, replay tools and their supported versions. Define how long old messages and old binaries remain eligible.
2. Add compatible handling and semantic fixtures to consumers. Test actual retained examples, null/absence, unknown enum behavior, deletes and tenant constraints.
3. Deploy consumer support with old producers still active; inspect decode failures, semantic mismatches and processing lag.
4. Enable the new producer for a small cohort only after required readers support its output. Preserve stable business identities and bounded replay/rollback paths.
5. Observe downstream outcomes rather than only publish success. A 200 response from the broker says nothing about a consumer's new enum branch.
6. Retire old paths only after the declared compatibility horizon closes and rollback/repair plans no longer require them.

Read the diagram

New consumer reads retained version 1 records → Compatible optional version 2 field is enabled → Old and new reader pairs are tested → Replay preserves original business event identity → Old support retires after the declared horizon

View full sizeOpen original

Schema, API and Event Evolution

Reset viewClose

Deployment order follows the tested compatibility pairs and retained replay horizon.

If a change is intentionally incompatible, use an explicit versioned contract or a new event type/topic with a defined migration plan. Running old/new streams can double-process business work. State whether both representations carry the same original business identity and how consumers avoid duplicate effects. Do not create a new charge simply because an adapter emitted a new transport envelope.

Preserve meaning through adapters and replay

An upcaster can transform W1 into R2's internal model when the transformation is defined and sufficient evidence exists. Adding display_label = absent is possible here. Inventing the currency for an old record that never stored it is not a safe schema fix. Quarantine/report unsupported input and recover evidence through the owning system instead of fabricating a plausible value.

Retain source event ID, aggregate version and delete semantics when adapting. Store adapter/version metadata separately for audit. If an adapter changes behavior, a reprocessing generation can identify the repair while the business effect still deduplicates the original event. Dedupe lifetime must cover the promised archive replay horizon.

A month-old v1 event replayed after v2/v3 deployments must be checked against the actual active reader, not merely a successful v2→v3 registration check. Adjacent compatibility is not automatically the same policy as checking all retained versions. See Confluent schema compatibility for its format-specific and transitive policy options.

Evolve HTTP APIs and workflow code too

For an API response, adding a field may still break strict consumers; removing or reinterpreting one is riskier. Specify version negotiation, error/unknown-state behavior and deprecation evidence. Track supported client versions and real use before removing a path. Do not assume every mobile client upgrades within a server deployment window.

For mutating API retries, compare canonical business intent across representations. If v1 and v2 both express the same 1,250 minor-unit order, retries under one scoped operation key must resolve to the same accepted resource. If v2 changes quantity or currency, it is a payload conflict or a new operation under policy. Blindly fingerprinting raw JSON bytes can treat harmless field-order/default changes as new meaning, while an over-aggressive normalizer can erase a real difference.

Durable workflow history may be replayed by new code long after its original worker disappeared. Preserve compatible decisions or apply the engine's supported versioning/migration approach. Deploying new activity code does not retroactively undo completed external effects. Schema migration, event compatibility and deterministic workflow replay have related but distinct boundaries.

Treat serialization rules as implementation-specific

For binary Protocol Buffers, field numbers and unknown-field handling have documented rules. Do not reuse removed field numbers, and review reserved names/numbers and JSON/text conversions. Binary round-tripping behavior cannot be assumed for every JSON adapter or generated-language enum branch. Use the Protocol Buffers language guide and test the actual toolchain. This lesson's JSON matrix is not a Protobuf or Avro compatibility checker.

Link storage cutover to the same version inventory

Changing a database representation can leave old readers/writers or replay jobs running. Preserve a clear authority and a tested compatible representation during the rollback window. A complete scan does not prove the newest concurrent changes or deletes were applied. A successful new consumer does not prove an old support script can interpret the new schema.

Use the Zero-downtime Schema Migration workshop for the concrete reader/writer matrix, conditional backfill, common-checkpoint validation and irreversible boundary. This shared lesson supplies the cross-interface contract; the workshop supplies the storage migration runbook.

Operate the rollout with a finite failure policy

Track unsupported versions, decode errors, semantic fixture failures, adapter quarantines, consumer lag and reprocessing outcomes by bounded version/type dimensions. Preserve secured examples that reproduce mismatches without logging unnecessary payloads. If one poison schema blocks a partition, the disposition policy must make the gap visible; silently skipping a financial event changes correctness.

A producer rollback is safe only if the old code can still read the accepted state and all required consumers remain supported. Once new-only meaning was accepted, rollback may require forward repair rather than just redeploying an old binary. Record this gate before activation, alongside the retention and deprecation decisions.

Check your understanding

Question: W2 adds an optional field but one old client rejects unknown JSON properties. Is this automatically forward compatible?

Question: A registry accepted v3 against v2. Does that prove a new consumer can replay retained v1 records?

Question: An upcaster emits a new event envelope for original E7. Should the payment effect receive a fresh business operation ID?

Transfer exercise and scoring

For API Design, Change Data Capture and Temporal, define the required version pairs, semantic fixtures, replay horizon, identity-preserving adapter and irreversible/deprecation gate. Award one point per concrete answer. Include a structurally valid but semantically wrong message before declaring the rollout safe.

Prerequisites: Idempotency and Delivery Semantics and Transactions and Isolation. The matrix, event, adapters and rollout are original teaching examples. Tests exercise this small JSON contract and retained fixtures, not a real schema registry or workflow engine.