Prerequisites and scoped previews
Events, identity and data contracts
Define keys, event/ingestion/availability times, schema evolution, dedupe and permitted use.
Labels, delayed outcomes and human annotation
Define label windows, maturity, censoring, annotation policy and disagreement.
What, when, why and how
Temporal and entity splits
- what
- A temporal split separates earlier training from later evaluation. An entity split holds out entire users/accounts/documents or other related groups.
- when
- Use temporal splits for future performance; use entity isolation when measuring generalization to unseen entities. Combine them when the deployment question needs both.
- why
- Randomly splitting related observations can reward memorization and future knowledge. A single universal split cannot answer every deployment question.
- how
- Write the deployment question first, then assign examples by decision time and grouping key. Apply horizon/embargo rules for overlapping labels and document which populations are held out.
Preprocessing and selection leakage
- what
- Leakage includes fitting transforms on evaluation data and repeatedly choosing models from a supposedly final test set.
- when
- Use these boundaries for imputation, scaling, vocabularies, target encoding, feature selection and hyperparameter tuning.
- why
- Even without explicit future columns, statistics or model-selection feedback can transmit test information into training.
- how
- Fit transformations inside the training fold, tune on validation only, then assess once on a held-out test. For target encoding, use appropriate out-of-fold/time-aware construction, not the row’s own outcome.
Contamination and overlapping evidence
- what
- Duplicates, near duplicates and shared source documents can cross nominal split boundaries.
- when
- Use grouping/content checks for copied posts, repeated purchases, document chunks, augmented images and evaluation prompts.
- why
- Different IDs do not establish independent evidence. A memorized answer or duplicate content can masquerade as generalization.
- how
- Keep parent/source/cluster identity, audit exact and approximate overlap, and state how unresolved similarity affects conclusions. Use a future or independently collected evaluation set where feasible.
Work a temporal split with a label horizon
Synthetic day numbers use a consistent UTC boundary. Training decisions occur before day 8, validation decisions from day 8 through before day 10, and test from day 10 onward. Each target observes outcomes for two days. To prevent training labels from extending into validation in this fixture, require training decision_day + 2 < 8. This is a conservative strict-boundary rule, not a universal embargo duration.
A at day 3 is train-eligible; B at day 5 is train-eligible; C at day 7 is purged from training because its label extends past the cutoff; D at day 8 is validation; E at day 10 is test. A model trained at day 8 must also require label availability by day 8, not just a nominal horizon.
If A and E are from the same customer, that may be acceptable for predicting future behavior of existing customers if past history is available. It is invalid evidence for a claim about unseen-customer generalization. Create a group-held-out evaluation for that question; declare how it changes population and sample size.
| Row / decision day | Label horizon ends | Fixture membership |
|---|---|---|
| A / 3 | 5 | train |
| B / 5 | 7 | train |
| C / 7 | 9 | purged |
| D / 8 | 10 | validation |
| E / 10 | 12 | test |
Catch a transform leak by hand
Training values are 0 and 2, so their fitted mean is 1. A validation value is 100. If an imputer is fit on all three rows, its mean becomes 34. A missing training value would then be imputed using validation information. Fit mean 1 on training; reuse that transform for validation/test without refitting.
Normalization and vocabulary learning need the same discipline. A pipeline that fits preprocessing per training fold is safer than preprocessing everything before cross-validation. Time-aware splits remain necessary; a correctly scoped scaler does not fix future labels or source duplicates.
Keep a final test set outside routine model selection. If an issue is diagnosed using that set and the model is changed, report that the set has become development evidence and obtain a suitable fresh assessment rather than claiming untouched test quality.
Read generalization evidence as hypotheses
Illustrative losses: training 0.10 and validation 0.60 suggest overfitting or a mismatch; they do not prove either. Check data/label/split compatibility before choosing regularization. Training 0.60 and validation 0.62 may suggest underfitting, noisy labels, poor optimization or inadequate signals. Inspect learning curves and controlled changes.
A model that performs well on validation but degrades on a later month may face distribution change, changed labels or serving skew. Compare input availability, preprocessing versions, score distributions and mature quality by cohort. Drift alone does not prove quality loss; stable distributions do not prove quality.
A smaller model, stronger regularization, better labels, more representative data or transfer learning are candidates under explicit evidence. Pick a discriminating experiment, change one important variable at a time, and respect evaluation selection boundaries.
Repair the split, not just the score
If duplicate documents cross the boundary, group them by source or duplicate cluster and rerun the experiment. If a future account status was joined into older predictions, reconstruct status as of availability and evaluate again. State which former result is invalid and why.
Version the split manifest with snapshot, grouping rules, cutoffs, horizon, purged rows, seed if relevant and transform-fit membership. A seed repeats a partition; it cannot make an invalid partition meaningful. Store counts and population coverage so repairs do not hide difficult examples.
Diagrams and relationships
This fixture purges day-7 training decisions because their two-day labels overlap the validation boundary.
Scroll horizontally on smaller screens, or enlarge the diagram.
Engineer, Senior and Staff follow-ups
These are teaching lenses. Each answer changes the scope of reasoning; they are not official employer leveling criteria.
Engineer: Why is day 7 purged?
Its two-day label ends on day 9, across the day-8 validation boundary. The fixture requires the training label end to be strictly before day 8, so it is not training-eligible.
Engineer: What mean should impute a missing validation value?
Use the training-fitted mean 1. Do not include validation 100 in fitting. Validation may reveal poor performance, but it cannot become a source for the fitted transform in this assessment.
Senior: Should every user appear in only one split?
It depends on the deployment question. For future interactions of known users, temporal evaluation with legitimate past history may be appropriate. For unseen-user performance, use group separation and report the changed target population.
Senior: How do you repair a contaminated final test?
Record the contamination and stop claiming independent final-test evidence. Revise grouping or source isolation, refit the development procedure and obtain an appropriate held-out assessment. Do not merely delete failing examples until a target score appears.
Staff: How do you standardize splits across many teams?
Standardize contracts, lineage and evaluation questions, then support task-specific split rules. Publish reference temporal/entity/duplicate fixtures and review exceptions. A single platform default can silently invalidate domains with different horizons or entities.
Staff: What migration risk follows from correcting leakage?
Offline scores may drop although validity improves. Communicate the invalid prior claim, recalibrate promotion criteria against clean evidence, keep serving safety gates and coordinate teams consuming the old benchmark. Treat benchmark lineage as an owned contract.
Practice and self-review
Your practice draft
Assign A–E using the stated cutoffs, compute training-only imputation, and explain whether this assessment measures future known-user or unseen-user quality.
Saved locally on this device. The worked answer does not score your draft. Existing lesson notes and completion are separate.
Progressive hints
Hint 1
Check decision time plus label horizon.
Hint 2
Keep transform fitting inside the training boundary.
Hint 3
State the entity overlap question separately from chronology.
Reveal worked answer guidance
Train A/B, purge C, validate D and test E. Fit mean 1 using 0/2 only. This alone measures neither a universal deployment population nor unseen-user quality: declare whether users overlap and whether that matches production. Add an entity-held-out assessment if new-user performance is the target.
Evidence to check in your answer
- Correct memberships and strict boundary.
- Availability and maturity checks included.
- Training-only transforms.
- Deployment question and duplicate/entity limitations stated.
For each item, record Missing, Prompted, Independent or Adaptive evidence. Adaptive means you can revise the reasoning under a changed constraint. Reading completion and readiness are separate.
Sources and verification scope
Original synthetic examples and reference calculations. Sources support narrowly scoped background; they do not establish production performance or independent chapter review.
Rules of Machine Learning ↗ · Baseline/infrastructure and training-serving consistency principles; dated examples are not current architecture claims.
Feast point-in-time joins ↗ · Historical feature retrieval reference; actual availability/revision contracts still need explicit modeling.
Primary background links checked during authoring on 2026-10-19. Arithmetic, temporal eligibility, maturity and split reference functions are checked by the bounded local fixture suite. SQL is explanatory, not a migrated production query. No GPU, external dataset, model API or training job is required.