ML system design interviews

The skeleton, and the calculator#

Walk every design through the seven steps above, in order, out loud. Two habits separate strong candidates: they ask questions before drawing, and they do arithmetic in front of the interviewer.

The arithmetic is easier than people fear. This is the entire feed-ranker envelope:

python3 --version   # standard library only, nothing to install

dau = 10_000_000                 # daily active users
feed_loads = 6                   # feed opens per user per day
candidates = 500                 # posts scored per feed open

requests_per_day = dau * feed_loads
scores_per_day = requests_per_day * candidates
peak_qps = requests_per_day / 86_400 * 3     # traffic is spiky; 3x the mean is a fair peak

ms_per_item = 0.05               # a small ranker, batched, on CPU
cpu_seconds = scores_per_day * ms_per_item / 1000
machines = cpu_seconds / 86_400  # one core gives 86,400 CPU-seconds per day

print(f"feed requests/day : {requests_per_day:,}")
print(f"items scored/day  : {scores_per_day:,}")
print(f"peak QPS          : {peak_qps:,.0f}")
print(f"scoring cores     : {machines:.0f} (single-core equivalents)")

feed requests/day : 60,000,000
items scored/day  : 30,000,000,000
peak QPS          : 2,083
scoring cores     : 17 (single-core equivalents)

Every input is an assumption, and that is the point: say each assumption out loud, and the interviewer will correct any they dislike. Thirty billion scores a day is the number that shapes the whole design — it is why the expensive model cannot see every post.

Design 1: a feed ranker#

Requirements to ask before drawing anything. What surface — home feed, reels, notifications? What does the business mean by "good" — taps, watch time, or longer-term return visits? How fresh must content be — is a 5-minute-old post eligible? What is the latency budget — a feed open should complete in about 200 ms. Cold start — what do brand-new users see?

Metric. Offline: ranking quality on held-out interactions, covered in ranking metrics. Online: the business metric via an A/B test, with guardrails such as reports and hides. Never present one without the other.

Architecture.

 OFFLINE
 interaction logs ─► label builder ─► training data ─► daily training ─► model store
       ▲                                                                    │
       │                                                                    ▼
 ONLINE│                                                          (models loaded by servers)
 user opens feed ─► candidate generation ─► feature fetch ─► ranker ─► business rules ─► feed
                    follows, popular,       (feature store)  scores     diversity,
                    similar-user posts:                      500 posts  freshness,
                    ~500 candidates                                     ads slots

Why two stages. Scoring millions of posts per request is impossible inside 200 ms. Candidate generation cheaply narrows millions to hundreds — typically with two-tower models, where user and post embeddings are compared by fast nearest-neighbour lookup. The expensive ranker then scores only those hundreds. Say this unprompted; it is the single most expected sentence in the round.

Data and training loop. Labels are implicit: taps, watch time, hides. Two honest problems to raise yourself. First, position bias — users tap what was shown at the top, so logs flatter whatever the old model ranked highly. Second, the feedback loop — the model learns from clicks it caused, which narrows what users see; see diversity and filter bubbles. Retrain daily; features must be computed identically offline and online, which is the reason feature stores exist.

Failure modes. Training-serving skew — a feature computed one way in training and another way live. Stale features after a pipeline delay. A celebrity post creating a hot key that hammers one cache shard. A new model that wins offline and loses online — which is why rollout goes through an A/B test at 1%, then 10%, as in A/B testing recommenders.

What interviewers probe. "Why two stages?" — cost arithmetic, above. "How do you know a new model is better?" — offline metrics gate the A/B test; the A/B test decides. "What breaks at 10x users?" — candidate generation and the feature store first; the ranker scales out horizontally.

Design 2: a UPI fraud detector#

A payments company wants to catch fraudulent transactions as they happen.

Requirements to ask. Decision must be inline — the payment waits on you, so the budget is roughly 100 ms end to end. What actions exist — allow, ask for an extra OTP, or block? What does a mistake cost each way? Blocking a genuine payment burns trust; missing fraud costs money directly. The action set matters more than the model: an extra-OTP middle option makes borderline scores survivable.

The envelope, spoken aloud. Assume 50 million transactions a day — about 580 per second on average, and several thousand at festival peaks. Assume fraud is 2 in 10,000 — that is 10,000 fraud attempts a day hiding in 50 million. Flag even 1% for step-up checks and you interrupt 500,000 genuine-looking payments daily. Those three numbers drive everything below.

Metric. Accuracy is meaningless at this imbalance — predicting "never fraud" scores 99.98%. Use precision and recall at the chosen threshold, and PR-AUC for model comparison, as covered in model evaluation and imbalanced data. Business metric: money lost to fraud plus genuine payments blocked, weighted by cost.

Architecture.

 payment request ─► feature fetch ─► model score ─► threshold policy
       │            device, amount     (< 100 ms total)     │
       │            history, velocity              ┌────────┼────────┐
       │            counters                     allow   extra OTP  block
       ▼                                                    │
 all decisions logged ─► analyst case review ─► confirmed labels ─► weekly retrain
                          (labels arrive days or weeks late)

Data and training loop. The hard part is labels. Fraud is confirmed by chargebacks and complaints, which arrive days or weeks later — so today's training data describes last month's fraud. Raise this yourself. Practical answer: retrain frequently on the freshest confirmed labels, keep fast-updating rule lists for brand-new attack patterns, and treat analyst review as a labelling pipeline, not an afterthought — see data labelling.

Failure modes. Adversarial drift: fraudsters probe the system and adapt, so the data distribution shifts because you deployed — the sharpest version of monitoring and drift. Festival spikes: Diwali-night traffic looks anomalous to a model trained on ordinary weeks, and blocking honest payments at scale that night is a headline. Feature-store timeout: decide in advance whether you fail open (allow) or fail closed (extra OTP), and say the trade-off out loud.

What interviewers probe. "Where do you set the threshold?" — from the two mistake costs, not from a metric curve alone; the researcher block derives it. "How do you retrain when labels arrive three weeks late?" — freshest-confirmed data plus rules for the gap. "Fail open or fail closed?" — there is no free answer; pick per amount, and small payments fail open.

Common mistakes#

Opening with the model. "I'd use XGBoost" as sentence one signals you skipped requirements. The model appears at step five.

No numbers, ever. One rough envelope, spoken aloud, outranks a beautiful diagram with no scale attached.

Designing for the wrong scale. Ask the scale first. A startup design with three components beats a planet-scale design the company will never need.

No monitoring story. A design that cannot detect its own failure is incomplete. End every design with what you watch after launch.

Silent thinking. The interviewer can only grade what you say. Narrate the trade-offs, including the ones you decide against.

Try it yourself#

Set a 45-minute timer and design "predict food delivery time" out loud, alone, through all seven steps. Speaking to an empty room feels ridiculous and is the single highest-value practice available — the round tests narrated thinking, and narrated thinking is trainable.

What to learn next#

• LLM system design interviews — the same round with a language model in the middle.
• A/B testing recommenders — how "is the new model better?" is actually decided.
• Monitoring and drift — the part of the design most candidates forget.