Prerequisites and scoped previews
Feature engineering and transformations
Create numeric/categorical/text features, defaults and fitted transformations without leaking validation data.
Scoped prerequisite preview. The complete chapter is planned; this preview does not count as authored coverage.
Point-in-time joins and feature availability
Use both event-time and actual availability when reconstructing historical decisions.
Training pipelines and orchestration
Own validated data snapshots, retries, checkpoints, lineage and budgeted work.
Scoped prerequisite preview. The complete chapter is planned; this preview does not count as authored coverage.
Reproducibility and experiment tracking
Record data/code/config/environment/seeds/artifacts and distinguish replayability from bitwise determinism.
Scoped prerequisite preview. The complete chapter is planned; this preview does not count as authored coverage.
What, when, why and how
Feature and transform parity
- what
- Parity requires matching feature definition, entity/context, availability, default semantics and transform version before comparing scores.
- when
- Use golden examples, shadow reads and replay when validating a model/feature release or diagnosing skew.
- why
- Offline and online can call the same transform on different source values. Conversely, equal scores can hide compensating input/transform errors.
- how
- Compare source IDs/status and contract versions first, then raw/transformed values and score outputs under declared numeric tolerances. Record reasons for missing/stale/default paths rather than comparing only successful rows.
Compatible bundles and golden fixtures
- what
- A bundle binds model input schema and ordered feature versions/transforms/defaults to the consuming model/policy release. Golden fixtures have independently checkable expected outputs.
- when
- Use bundle validation for promotion, rollback, feature semantic changes and data repair.
- why
- A dimension/type match does not establish units or meanings. Duplicating an implementation in a test can repeat the same error without an oracle.
- how
- Pin exact versions and include numeric, boundary, missing, stale, late-event and unauthorized cases. Use a small independent SQL/Python calculation where possible and state what is illustrative or design-only.
Tolerance and runtime differences
- what
- Floating-point, encoding, categorical ordering and preprocessing differences can create score variation. Some invariants are exact; others need a justified tolerance.
- when
- Use this across batch/stream languages, serialization changes and serving runtimes.
- why
- A huge tolerance can hide semantic errors, while unjustified bitwise equality can reject valid numeric variation. A random seed alone does not prove determinism.
- how
- Require exact identity/order/status/schema checks; use a scoped numeric tolerance for declared floating-point operations. Report environment and scale. A count-to-rate or /10-to-/100 change is not ordinary rounding noise.
Compute a golden feature and score
Under scale-10-v1, raw feature 40 transforms to 4. The synthetic fixed formula score=2*x+1 returns 9. An online implementation using /100 produces .4 and score 1.8; that is a semantic mismatch, not a harmless floating-point deviation. No model is trained in this calculation.
For raw 10 the same contract gives transformed 1 and score 3. A missing/stale raw default 0 gives score 1 only under the consuming contract with its explicit status flag; deletion/authorization denial returns no usable value and must not be silently scored as zero.
The reference compares identical source/status/version/transform and a tolerance-bound value. Golden equality on two inputs does not prove a production pipeline correct at scale; add representative and independent failure coverage.
Trace skew from source to output
Given an old prediction ID, reconstruct its tenant/entity/generation and compatible feature/model bundle, then inspect the selected serving-activation interval. Compare producer/schema versions, materialization rejection/lag, raw values, defaults, transform outputs and final scores in that order.
An offline query at now() and a cached current read are not an old serving replay. Align the question and cutoffs first. If activation history or old permitted inputs were not retained, disclose the limit rather than inventing a successful replay.
Migrate units with explicit versions
Publish v2 under a new feature contract/namespace; do not overwrite v1 values with a rate while old models still expect a count. Dual-materialize bounded samples or shadow traffic, verify parity within each version and evaluate the intended cross-version quality difference.
Switch an approved compatible model-feature reference only after data/coverage/resource gates. Keep the previous permitted contract/bundle available for rollback. A rollback may require restoring feature namespaces and transforms, not only model weights; deletion/current access still applies.
Avoid false reassurance
An endpoint health check stays green while defaults jump from 1% to 40%. Track missing/stale/default rates, raw/normalized value distributions, source coverage and mature outcome slices. Restoring data freshness may be more appropriate than retraining a model.
A shared library can prevent some duplicate transform code but does not make source windows, timestamps, filters, schemas or online materialization identical. Validate both shared logic and each path’s input contracts.
Diagrams and relationships
Identity and availability are exact contracts; the toy numeric score supplies an independent hand-checkable output.
Scroll horizontally on smaller screens, or enlarge the diagram.
Engineer, Senior and Staff follow-ups
These are teaching lenses. Each answer changes the scope of reasoning; they are not official employer leveling criteria.
Engineer: What does /100 change in the example?
It changes feature meaning/scale: 40 becomes .4 instead of 4, and score becomes 1.8 instead of 9. Reject the incompatible transform rather than increasing numeric tolerance.
Engineer: Can a deleted feature use the ordinary zero default?
Not by silently treating access denial as missing data. The fixture returns no value for deletion and refuses to score it; the domain defines any permitted fallback outside the revoked data path.
Senior: Which parity checks should be exact?
Entity/tenant/generation, feature/model/transform identities, feature order and missing/default status should follow exact contracts. Numeric comparisons may use a declared tolerance appropriate to the operations, but not to conceal semantic changes.
Senior: Why compare input values before scores?
Compensating errors can produce equal scores, and a score mismatch may originate in source selection rather than the model. Trace raw source, status, transforms and order before output diagnosis.
Staff: How do you share transformations across teams?
Publish semantic/version contracts, tests and support windows, then validate each path’s data availability and ownership. Coordinate model/feature migrations and deprecation; common code alone does not establish equivalent input populations.
Staff: What does rollback need after a feature migration?
The permitted previous feature namespace, transform/default rules, compatible model/policy bundle and lineage needed to validate reads. If prior data is revoked or unavailable, use an approved fallback rather than pretending rollback restores legal or historical access.
Practice and self-review
Your practice draft
Compute both transforms and scores, then define a parity incident drill covering source identity, defaults, version mismatch and a deleted entity.
Saved locally on this device. The worked answer does not score your draft. Existing lesson notes and completion are separate.
Progressive hints
Hint 1
Check input/source/status contracts first.
Hint 2
Use raw 40 and a fixed scalar function as an independent oracle.
Hint 3
Separate denied/deleted from ordinary missing.
Reveal worked answer guidance
Correct /10 yields x=4, score 9; /100 yields .4, score 1.8. Exact contract/source/status checks catch incompatibility, then a justified numeric tolerance checks values. Missing/stale may default under policy; deleted/denied data supplies no model input. Trace materialization and activation before blaming weights.
Evidence to check in your answer
- Exact identity/order/status boundaries.
- Hand-derived golden output.
- No tolerance masking of unit changes.
- Compatible rollback and access caveats.
For each item, record Missing, Prompted, Independent or Adaptive evidence. Adaptive means you can revise the reasoning under a changed constraint. Reading completion and readiness are separate.
Sources and verification scope
Original synthetic examples and reference calculations. Sources support narrowly scoped background; they do not establish production performance or independent chapter review.
Rules of Machine Learning ↗ · Baseline/infrastructure and training-serving consistency principles; dated examples are not current architecture claims.
Feast point-in-time joins ↗ · Historical feature retrieval reference; actual availability/revision contracts still need explicit modeling.
Primary background links checked during authoring on 2026-10-19. Arithmetic, temporal eligibility, maturity and split reference functions are checked by the bounded local fixture suite. SQL is explanatory, not a migrated production query. No GPU, external dataset, model API or training job is required.