Prerequisites and scoped previews
Sampling, imbalance and observation bias
Explain selection/exposure bias, negatives, weighting and what sampling changes.
Embeddings and representation contracts
Define embedding training objective, normalization, dimension and model/index compatibility.
Candidate retrieval and ANN indexes
Separate eligible-corpus recall from latency/memory and filtering effects.
What, when, why and how
Retrieval and ranking separation
- what
- Retrieval selects a feasible candidate set; ranking predicts or orders utility within that set. The final slate policy applies additional constraints.
- when
- Use stages when evaluating every catalog item with the richest model is too expensive for the request budget.
- why
- A perfect ranker cannot return a relevant item that retrieval omitted. A high retrieval score is not necessarily a calibrated probability of user satisfaction.
- how
- Start with a complete popularity baseline, add candidate sources and a bounded retrieval limit, then rank unique eligible candidates. Measure candidate recall separately from final ranking quality and log which stage excluded an item.
Ranker objective and input contract
- what
- A ranker consumes candidate/user/context features and produces a score or multiple outcome estimates under declared time and version semantics.
- when
- Use a richer pairwise interaction model after retrieval, where the candidate count is bounded.
- why
- A model trained to predict clicks can reward clickbait; a feature computed after exposure leaks future outcomes. A loss is a training objective, not the complete product policy.
- how
- Compare a simple linear/tree scorer with an optional richer sequence/joint-interaction model. Define input groups and defaults, target/negative construction, loss and calibration needs. Use ablations and mature held-out outcomes to justify complexity.
Slate constraints and feedback
- what
- A slate is an ordered collection. Creator caps, deduplication, safety and diversity can change the order produced by independent item scores.
- when
- Use explicit slate policy when output quality depends on combinations and repeated exposure, not only individual scores.
- why
- A greedy cap can improve diversity while reducing unconstrained relevance. Logging only final clicks hides which candidates were eligible or excluded.
- how
- Define a deterministic policy/version and trace rejected items. Log decision, candidate provenance where permitted, final positions and actual exposure. Evaluate utility and guardrails after the policy, not only before it.
Trace a five-candidate slate
Synthetic scores: A=.9 and B=.8 from creator C1, C=.7 from C2, D=.6 from C3; E=.5 from C4 is ineligible. Deduplicate IDs, remove E and sort by score then item ID. Unconstrained top-3 is A,B,C. Under a cap of one video per creator, the output is A,C,D.
The cap skips B despite its higher score than D. This is an authored policy choice, not proof of better satisfaction. Compare creator coverage, relevance and long-term outcomes. If the requested size is four but only three creators remain, return three rather than bypass the cap silently.
If two candidate sources both return A, rank A once. Preserve a set of source tags rather than repeating its training/evaluation weight. Apply an authoritative eligibility check immediately before display to catch changes after retrieval.
Prioritize features with a testable hypothesis
Use item metadata, pre-request user history and request context as initial signal groups. Item language/topic can support matching; history can capture prior preference; context can select the placement and active session. Unknown history needs a declared default and missing indicator.
Reject same-request watch completion and post-exposure likes as inputs to that request. A permitted earlier event can still be unavailable at serving time. Compare an item-only model, then add history/context with held-out evaluation; a higher offline score on a leaked dataset is not an ablation win.
| Group | Representation / freshness | Verification |
|---|---|---|
| Item | Versioned topic/language/content vector; catalog availability | New-item and revoked-item slices. |
| User history | Pre-request sequence or aggregates; bounded lookback | Empty history and late events. |
| Context | Placement/device/session fields available at request | Missing/unknown categorical values. |
| Rejected input | Current exposure’s future watch duration | Deliberate leakage fixture must reject it. |
Budget expensive work and degraded modes
Assume retrieval emits at most 200 candidates and the expensive scorer handles at most 100. A cheap stage selects that 100 under a versioned rule. The end-to-end candidate budget is visible: a relevant item lost at this stage cannot be recovered by the expensive ranker.
Batch feature reads and score work under a request deadline; bound queues and cancel/stop work that no longer serves the request. If ranker scoring fails, return the eligible baseline order under the same slate policy. Avoid blocking on analytics delivery; record outbox/retry semantics for permitted telemetry separately.
A single simple ranker is preferable when its quality meets requirements and extra stages add maintenance or tail-latency cost without validated benefit. A sequence model is an optional hypothesis requiring data/resources, not a default interview badge.
Connect the slate to trustworthy evidence
A returned response is not proof every item was viewed. Record actual eligible impressions with decision ID, item, position, time, model/index/policy bundle and event identity. Later interactions reference exposure; pending windows stay pending.
Random negatives from the corpus are useful modeling samples under explicit assumptions, not verified disinterest. Position and prior policy influence observed clicks. An exploration or controlled experiment requires its own support, assignment and guardrail contract; logging unshown scores does not reveal their counterfactual outcomes.
Diagrams and relationships
The slate policy acts after scores; its exclusions must remain explainable.
Scroll horizontally on smaller screens, or enlarge the diagram.
Engineer, Senior and Staff follow-ups
These are teaching lenses. Each answer changes the scope of reasoning; they are not official employer leveling criteria.
Engineer: Why is B absent despite score .8?
A and B share C1, and the declared cap admits one item per creator. The deterministic greedy policy keeps A, skips B, then selects C/D. This is a policy effect, not a scoring error.
Engineer: What if E becomes eligible after ranking?
The current response uses a defined decision snapshot and final policy check. It need not backfill E after a completed selection; a later request can include it. Do not mutate historical candidate logs to match current eligibility.
Senior: How do you tell retrieval loss from ranking loss?
Inspect eligible relevant IDs before retrieval, in retrieved candidates, after cheap selection and in the final slate. Compute stage-specific recall and reranking metrics under the same judgments, retaining denominators and exclusion reasons.
Senior: What does a ranker outage change?
Switch to the declared eligible baseline under the same safety/creator policy and deadline. Mark the fallback bundle in telemetry, measure quality and error budgets, and avoid treating fallback output as a successful learned score.
Staff: How do teams negotiate conflicting slate goals?
Assign owners to utility, safety and creator/exposure constraints; define measurable trade-offs and change control. Version the policy independently of scores but validate the full bundle. A globally fixed weight cannot resolve every product conflict without explicit priorities.
Staff: How should training respond to repeated policy-driven exposure?
Preserve exposure and selection provenance, audit population coverage and distinguish observed outcomes from unobserved alternatives. Explore permitted controlled exposure with safeguards; document support and weighting assumptions rather than claiming logs automatically remove bias.
Practice and self-review
Your practice draft
Build the top-3 slate with a creator cap, trace B/E exclusions and describe a scorer timeout. Name three signal groups and one rejected future feature.
Saved locally on this device. The worked answer does not score your draft. Existing lesson notes and completion are separate.
Progressive hints
Hint 1
Deduplicate and establish eligibility before counting candidates.
Hint 2
Trace score ordering separately from the creator constraint.
Hint 3
A fallback keeps the same invariants and a distinct telemetry reason.
Reveal worked answer guidance
The capped slate is A/C/D. B is skipped because C1 is represented; E is ineligible. On timeout use an eligible simple order with the same cap, recording degraded mode. Use available metadata, prior history and context; reject current-exposure outcomes as inputs. Evaluate final policy quality and stage-specific loss separately.
Evidence to check in your answer
- Deterministic cap and shortfall behavior.
- Separate stages and metrics.
- Feature availability and leakage check.
- Exposure/feedback limitations and ownership.
For each item, record Missing, Prompted, Independent or Adaptive evidence. Adaptive means you can revise the reasoning under a changed constraint. Reading completion and readiness are separate.
Sources and verification scope
Original synthetic examples and reference calculations. Sources support narrowly scoped background; they do not establish production performance or independent chapter review.
Deep Neural Networks for YouTube Recommendations ↗ · 2016 publication page/abstract inspected for candidate-generation versus ranking separation only. Full PDF could not be retrieved. No current YouTube architecture or full-paper review claim.
Google recommendation systems overview ↗ · General recommendation framing; original candidate/slate examples and workload assumptions.
Primary background links checked during authoring on 2026-10-19. Arithmetic, temporal eligibility, maturity and split reference functions are checked by the bounded local fixture suite. SQL is explanatory, not a migrated production query. No GPU, external dataset, model API or training job is required.