Question Breakdowns

Recommendation Serving

Serve personalized recommendations to 100 million users through bounded retrieval, ranking and policy stages with measurable fallback behavior.

A AceYourInterviews editorial · 8 min study

Serve personalized recommendations to 100 million users through bounded retrieval, ranking and policy stages with measurable fallback behavior. This worked design targets SDE-3; the scale figures below are exercise assumptions, not claims about a named company's implementation.

Clarify the prompt

Ask the interviewer	Decision to record
What product action is optimized, and what are safety/diversity guardrails?	Clicks alone may harm the product.
What latency, freshness and availability targets apply?	Define user-visible serving separately from training.
Which behavior data is permitted and when do deletion/opt-out changes take effect?

Requirements and scope

The core deliverable is: Serve personalized recommendations to 100 million users through bounded retrieval, ranking and policy stages with measurable fallback behavior. The first design implements the APIs below and the state constraints in the data model. Agree the latency, freshness, retention and failure policy before adding infrastructure. Every successful acknowledgment must have an explicit meaning: accepted intent, committed state or completed external work. These are separate milestones.

Capacity and bottleneck reasoning

Assume 100 million daily users requesting ten feeds/day: one billion requests/day, about 11,574/s average and 116,000/s at 10x peak. Ranking 500 candidates/request at peak means 58 million candidate evaluations/s; preselection is mandatory.

Keep average and peak demand separate. Add replication, indexes, retries and safety headroom to raw arithmetic before selecting capacity. Load-test the busiest ownership key and recovery traffic as well as uniform happy-path throughput.

APIs and data model

GET /recommendations?cursor=... -> items,requestId,modelVersion
POST /interactions {eventId,impressionId,action} -> accepted
POST /models/{version}/canary {cohort,share} -> rollout status

Contract / pseudocode

interaction(event_id,user_id,item_id,event_time,impression_id,action)
feature(entity_id,feature_version,as_of); embedding(entity_id,model_version,vector)
model(version,feature_schema,policy); exposure(request_id,item_id,position,model_version)

Authorize resource ownership at the API boundary; do not trust IDs or state supplied by a client. Bind retry identity to the caller and payload for mutating operations. Reject an incompatible replay instead of returning an unrelated stored result.

Component relationships

Recommendation Serving: directed relationships and authoritative boundaries — Recommendation Serving: state boundaries

Step-by-step design

1. Establish the authority

Log exposures as well as interactions so training distinguishes ignored items from unseen ones. Build time-correct labels and features; split evaluation by deployment time and verify that future outcomes do not leak into features.

2. Make retries and concurrent work safe

Retrieve candidates from followed entities, similarity, popularity and recent activity under per-source time budgets. Deduplicate, filter blocked/deleted items, fetch a bounded feature set and rank only the remaining bounded candidates.

3. Scale and recover the critical path

Use model and feature-schema versions together. Apply diversity and safety constraints after ranking, emit exposure identity and fall back to permitted popularity/followed content if feature or model serving exceeds its deadline. A fallback must preserve policy filters.

Worked request and concurrency example

For one feed request, retrieval returns 300 follow candidates, 300 ANN candidates and 100 trending candidates. After deduplication and eligibility checks, rank at most 500 within an 80 ms ranking budget of a 200 ms illustrative total. A feature store outage triggers a baseline model using available stable features, then a popularity fallback if necessary. The response records the actual strategy so experiment analysis does not attribute fallback results to the full model.

Failure and recovery walkthrough

A new model improves offline clicks but repeatedly recommends one narrow category. Canary by stable user cohort and monitor long-term engagement, diversity, safety and latency, not just aggregate CTR. Roll back the model/feature pair when guardrails regress.

Recommendation Serving: one concrete failure or state-transition sequence — Recommendation Serving: concrete trace

Decisions and alternatives

Decision	Trade-off
Alternative 1	Precomputed feeds reduce online cost but age quickly. Online ranking adapts to context at higher latency and feature-serving complexity.
Alternative 2	ANN retrieval improves scale but may miss relevant items. Measure retrieval recall separately from ranking quality; a strong ranker cannot recover a candidate that was never retrieved.

Operational contract

Monitor end-to-end/stage p99, fallback rate, feature freshness, exposure completeness, candidate-source recall and cohort quality. Test new users/items, deleted content, model timeout and experiment assignment stability.

Choose alert thresholds from the agreed user-facing objective. Assign an owner for stuck work and a replay/repair procedure that preserves the original business identity. Bound queue depth, concurrency and recovery throughput so an outage does not produce an uncontrolled catch-up burst.

Foundation, senior and staff depth

Foundation: explain the user action, model, first complete request and the correctness constraint without relying on a list of technologies.
Senior: defend this lesson's partition/coordination boundary, capacity estimate and failure recovery; explain the cost of the rejected alternative.
Staff: identify failure domains and operational ownership, stage a rollout with measured guardrails, and state when rollback stops being safe.

Practice before revealing the answer

Spend 45 minutes: 5 clarifying scope, 7 on capacity/APIs/state, 12 on the first architecture, 15 on the concurrency and failure examples, and 6 on trade-offs and operations. Draw your own design before comparing it with this chapter.

Question: How do 100 million users change the architecture if most are inactive?

Show answer and explanation

Answer: Capacity follows active requests, candidate counts and event volume, not registered-user count alone. Separate DAU, peak RPS and per-stage work, then partition feature and embedding storage according to access patterns.

Verification and rollout

Reproduce the worked example using two concurrent actors or a repeated request where applicable. Inject the stated failure between the durable transition and its acknowledgment; verify the resulting state and user-visible outcome. For read/ML paths, test missing or stale dependencies and verify the documented fallback still enforces permissions. Use a small canary, compare correctness and user-facing latency with the baseline, and roll back when those guardrails regress. Do not use an average throughput number as evidence that a tail-latency target is met.

Primary technical reference

Official documentation for the mechanism used in this exercise. This lesson is an original interview design, not a description of undisclosed company internals.

Before this design, review these mechanisms and identify which owns the critical state transition:

Use the Interview Playbook to adapt the depth and timing, then return to the question breakdown overview.