Prerequisites and scoped previews
Task families and label semantics
Distinguish classification, regression, ranking, retrieval, forecasting and generation.
Scoped prerequisite preview. The complete chapter is planned; this preview does not count as authored coverage.
Events, identity and data contracts
Define keys, event/ingestion/availability times, schema evolution, dedupe and permitted use.
What, when, why and how
Label definition, window and maturity
- what
- A label contract names the target, attribution unit, observation horizon, maturity delay, provenance and revision policy.
- when
- Use it for supervised learning, feedback ingestion and final test eligibility.
- why
- An outcome that has not happened yet is different from a verified negative. Treating new examples as negatives creates systematic bias.
- how
- For a seven-day purchase target, join exposure to an attributable purchase within seven days. Add an explicit ingestion allowance. Evaluate only when the horizon and allowance have passed and the source is complete; otherwise label pending.
Direct, proxy and adjudicated evidence
- what
- Direct outcomes, user reports and reviewer decisions are different evidence sources, each with potential error and selection.
- when
- Use these when combining sparse expert labels with abundant behavioral signals.
- why
- A report may be malicious or selective; a click may indicate curiosity rather than satisfaction. Calling every source ground truth hides assumptions.
- how
- Store source and confidence/provenance separately from the target. Audit disagreement and selected versus unselected cases, and validate whether a proxy predicts the actual product outcome.
Human review and revision
- what
- Annotation is an operational process with policy versions, queues, reviewer disagreement, quality checks and appeals.
- when
- Use it when targets require judgment or decisions can be corrected later.
- why
- A policy change can redefine the target; repeated labels do not automatically represent independent training examples.
- how
- Use versioned guidelines and adjudication rules, sample reviewer agreement with appropriate uncertainty, record immutable revisions, and choose one eligible revision per example at a declared snapshot cutoff.
Work a maturity policy
Assume seven calendar days after exposure plus one day ingestion allowance, all in UTC. At a dataset cutoff of October 10 12:00, exposure E1 from October 1 12:00 is mature. E2 from October 5 12:00 is pending until October 13 12:00. These are synthetic exercise timestamps.
E1 has an attributed purchase on October 3, available October 4: label positive. E2 has no purchase yet: pending, not negative. An older E3 exposure from October 1 with no purchase is negative only if the outcome feed is complete through its window and the allowance has elapsed. A source outage makes it unknown.
A late purchase correction available October 11 is absent from the October 10 serving-state snapshot. A later corrected-truth snapshot may revise the label. Pin the label revision and snapshot identity so the difference is explainable.
| Example | Window / source state at cutoff | Label |
|---|---|---|
| E1 | Seven days + allowance passed; attributable purchase available | 1 |
| E2 | Window still open | pending |
| E3 | Mature; source complete; no attributable purchase | 0 |
| E4 | Mature in wall-clock time; outcome feed incomplete | unknown |
Do not train only on easy-to-observe cases
A moderation queue sees reported posts and model-selected posts. These are not a random sample of all eligible content. Appeals disproportionately involve people motivated and able to appeal. Annotator agreement on the queue says little about unseen content prevalence.
Create a permitted audit sample from the target population with known inclusion rules. Track selection propensity where usable, but do not claim weighting can recover groups that have zero chance of observation. Report limitations when broad random labeling is unaffordable.
For purchases, last-touch attribution is one explicit policy, not the unique truth about causal influence. Exposure assignment, actual display and purchase are separate records. A model trained on purchases after recommendations cannot infer what would have happened without showing them.
Build a label runbook
A label revision references prediction/exposure, source record, policy version, observed and available times, and a status such as pending, provisional, adjudicated or revoked. Separate accepted training labels from raw reports. Retain only permitted evidence and redact sensitive content from routine logs.
If guidelines change, create a new policy version, assess whether older examples remain comparable and rerun affected slice evaluation. If reviewers disagree, route ambiguous examples for adjudication or represent uncertainty under an explicit learning policy. Do not erase disagreements to improve apparent agreement.
A simple manually adjudicated dataset may be preferable to weak-label complexity for an early baseline. Weak supervision or self-supervised representation learning can use additional data, but requires a distinct validation strategy and does not make proxy labels accurate by definition.
Budget labels for training and evaluation separately
Assume a budget of 1,000 adjudications/week. A proposed allocation of 800 uncertain training cases plus 200 randomly selected audit cases answers two different questions. The uncertainty queue may improve learning efficiency; the audit queue estimates behavior on its declared sampling population. Neither count guarantees a precise rare-event estimate.
If the audit sample contains zero positives, do not infer zero prevalence or perfect recall. Report the sample and uncertainty limitation, consider additional stratified sampling with known inclusion probabilities, and preserve a representative assessment process. Reviewer throughput and adjudication delay constrain when quality evidence will mature.
If a policy update makes 100 prior labels ambiguous, flag those revisions and rebuild affected snapshots under a new version. Keep a test asserting that older snapshots retain their pinned revisions. This cost and lineage discipline is often more valuable than adding a more complex model to inconsistent targets.
Diagrams and relationships
Time flows downward. Maturity requires the observation window, allowance and source completeness; a later correction creates a new revision.
Scroll horizontally on smaller screens, or enlarge the diagram.
Engineer, Senior and Staff follow-ups
These are teaching lenses. Each answer changes the scope of reasoning; they are not official employer leveling criteria.
Engineer: Is no purchase a negative?
Only after the declared attribution window and allowance, with adequate source completeness and eligibility. Before then it is pending; under an outage it may remain unknown.
Engineer: Can two label revisions be two independent examples?
Not for the same prediction under this contract. Select a single eligible revision at the snapshot cutoff and retain history for audits. Otherwise corrected examples receive extra weight unintentionally.
Senior: What if adding the latest week improves validation accuracy?
Check whether the latest week contains immature examples labeled zero and whether it leaks into validation. Compare cohort maturity, feed completeness and policy versions before interpreting the metric change.
Senior: How do you use user reports safely?
Keep them as a source with provenance and abuse controls, audit them against adjudicated outcomes, and distinguish their sampling from population labels. A report can be an input available after posting, but cannot be used for a pre-post decision.
Staff: How do two teams reconcile different definitions of success?
Version target definitions and attribution separately; agree on the product decision each serves. Shared raw outcomes can support different targets, but evaluations must name the policy and window. Do not average incompatible labels into a supposedly universal truth.
Staff: How do you reduce label cost without losing evaluation credibility?
Prioritize representative audit coverage and high-uncertainty cases for different purposes. Active learning can help training, while evaluation still needs a defensible target-population sample or justified weighting with uncertainty. Assign ownership for taxonomy, review quality and revisions.
Practice and self-review
Your practice draft
At October 10 12:00, build labels for E1–E4. Then describe what changes when a correction becomes available on October 11.
Saved locally on this device. The worked answer does not score your draft. Existing lesson notes and completion are separate.
Progressive hints
Hint 1
Compute horizon plus allowance for every example.
Hint 2
Wall-clock maturity does not prove the source is complete.
Hint 3
Keep October 10 and corrected October 11 snapshots distinct.
Reveal worked answer guidance
E1 is positive, E2 pending, E3 negative under complete-source assumptions, and E4 unknown. The October 11 correction creates a new revision; it must not silently mutate an already published October 10 snapshot. A corrected-truth analysis can use it under a new snapshot and documented replay goal.
Evidence to check in your answer
- Maturity and availability both checked.
- Missing and negative are distinct.
- Source/provenance and attribution declared.
- Correction lineage and privacy boundaries preserved.
For each item, record Missing, Prompted, Independent or Adaptive evidence. Adaptive means you can revise the reasoning under a changed constraint. Reading completion and readiness are separate.
Sources and verification scope
Original synthetic examples and reference calculations. Sources support narrowly scoped background; they do not establish production performance or independent chapter review.
Rules of Machine Learning ↗ · Baseline/infrastructure and training-serving consistency principles; dated examples are not current architecture claims.
Feast point-in-time joins ↗ · Historical feature retrieval reference; actual availability/revision contracts still need explicit modeling.
Primary background links checked during authoring on 2026-10-19. Arithmetic, temporal eligibility, maturity and split reference functions are checked by the bounded local fixture suite. SQL is explanatory, not a migrated production query. No GPU, external dataset, model API or training job is required.