ML theory questions, with answers

Setup#

Questions 6–15. Two of them are best answered with ten lines of NumPy, so:

pip install numpy

Both demos run on CPU in under a second. Outputs were produced with NumPy 2.4; the same seeds give the same numbers on any recent version.

Q6. Explain the bias-variance trade-off#

Bias is error from a model too rigid to capture the pattern. Variance is error from a model so flexible it captures the noise. Total error carries both, plus noise nobody can remove — so shrinking one usually grows the other, and you are choosing where to sit.

This is more convincing run than recited. Fit polynomials of increasing flexibility to fifteen noisy points from a sine curve:

import numpy as np

rng = np.random.default_rng(7)


def sample(n):
    x = rng.uniform(0, 1, n)
    return x, np.sin(2 * np.pi * x) + rng.normal(0, 0.25, n)   # true curve + noise


x_train, y_train = sample(15)      # small training set, like most real problems
x_test, y_test = sample(200)       # large held-out set, so test error is stable

print("degree   train error   test error")
for degree in (1, 3, 10):
    coef = np.polyfit(x_train, y_train, degree)
    train = np.mean((np.polyval(coef, x_train) - y_train) ** 2)
    test = np.mean((np.polyval(coef, x_test) - y_test) ** 2)
    print(f"{degree:>6}   {train:>11.3f}   {test:>10.3f}")

degree   train error   test error
     1         0.146        0.284
     3         0.023        0.147
    10         0.016       14.636

Degree 1 is a straight line through a sine wave: high error everywhere — that is bias. Degree 10 threads through every noisy point: near-zero training error, test error a hundred times worse — that is variance. Degree 3 sits in the valley between them. The interview sentence: "training error only ever falls with flexibility; test error is U-shaped, and I model-select for the bottom of the U." The formal decomposition is Q16.

Q7. L1 versus L2 regularisation — what is the practical difference?#

Regularisation adds a penalty on large weights to the training objective, taxing complexity. L2 penalises squared weights and shrinks everything smoothly towards zero, but rarely to zero. L1 penalises absolute values and drives many weights exactly to zero — it performs feature selection while training.

So: L1 when you suspect most features are irrelevant and want a sparse, explainable model. L2 as the default, especially with correlated features, where L1 arbitrarily keeps one and zeroes its twin. The geometric and mathematical reason L1 zeroes weights is Q17. Both appear in practice in linear-regression and logistic-regression.

Q8. Precision versus recall — and when do you optimise for which?#

Precision: of everything the model flagged, what fraction was truly positive. Recall: of everything truly positive, what fraction the model flagged. They trade off against each other through the decision threshold.

Optimise recall when a miss is expensive: cancer screening, where a missed case can be fatal and a false alarm costs one more test. Optimise precision when a false alarm is expensive: blocking payments as fraud, where each false block is a furious customer. Here is Q4's trap made concrete:

import numpy as np

y_true = np.array([0] * 950 + [1] * 50)     # 5% fraud, like a real payments table
y_pred = np.zeros(1000, dtype=int)          # a "model" that always says "not fraud"

accuracy = (y_pred == y_true).mean()
caught = ((y_pred == 1) & (y_true == 1)).sum()
print(f"accuracy: {accuracy:.1%}")
print(f"frauds caught: {caught} out of {y_true.sum()}")

accuracy: 95.0%
frauds caught: 0 out of 50

95% accurate, zero recall. Name the summary metrics too: F1 (the harmonic mean of precision and recall), PR-AUC (area under the precision-recall curve, threshold-free) for rare positives, and ROC-AUC when classes are balanced. Full treatment: model-evaluation.

Q9. How does cross-validation work, and when is one split not enough?#

k-fold cross-validation cuts the data into k parts, trains k times — each time holding out a different part for scoring — and averages the k scores. One split gives you a single noisy estimate; five or ten folds give a mean and a spread.

When: small datasets, where one split's luck dominates, and any time you are comparing models or tuning. When not: huge datasets, where one split is already stable and k training runs cost real money. The honest extra: with time-ordered data, random folds leak the future into training — use time-based splits instead. That sentence has ended interviews well. See train-test-split.

Q10. Logistic regression or a tree ensemble — how do you choose?#

Start with what the business needs from the model beyond accuracy. Logistic regression gives calibrated probabilities, coefficients a regulator can read, and works well when effects are roughly additive. Tree ensembles like random forests and XGBoost capture interactions and non-linear effects without feature engineering, and usually win on raw accuracy for tabular data.

The strong answer names a decision rule: regulated or high-explainability settings, or tiny datasets — logistic regression, possibly with hand-built interaction features. Accuracy-first tabular problems — gradient boosting, and keep the logistic model as the baseline that catches leakage. Saying "I would build both; the simple one is my sanity check" reads as experience.

Q11. What is the difference between bagging and boosting?#

Both build many weak models and combine them; they differ in how the models relate. Bagging trains models independently, in parallel, each on a random resample of the data, and averages them — variance goes down. Random forest is bagging plus random feature choices. Boosting trains models one after another, each focused on the mistakes of the ones before, and sums them — bias goes down. XGBoost is the standard implementation.

The trade-off line: bagging is robust and hard to overfit with more trees; boosting is more accurate when tuned and more able to overfit, so it leans on early stopping and learning-rate shrinkage.

Q12. Your classes are 99-to-1 imbalanced. What do you do?#

First, refuse to fix what may not be broken: if the model ranks well (good PR-AUC), the "problem" may be the metric, not the data. Then, in order of preference: pick metrics that see the rare class (Q8); use class weights, which most libraries support and which cost nothing; adjust the decision threshold to the business's cost ratio; resample only after those — oversampling duplicates rare examples, undersampling throws data away, and both distort probabilities.

The differentiating sentence: resampling changes the base rate your model believes in, so its probability outputs must be recalibrated before anyone treats them as probabilities.

Q13. How do you handle missing values, and what can go wrong?#

First ask why they are missing, because the mechanism decides the fix. If a sensor dropped values at random, filling with the median is fine. If missingness carries meaning — income left blank by people with low income — then the blank is information: add a "was missing" indicator column alongside any fill.

What goes wrong: computing the fill value on the whole dataset before splitting. That leaks test-set statistics into training — leakage again (Q5). Compute fills on the training split only, inside the pipeline. Deleting rows is honest but can silently delete a biased slice of your data; say you would check what remains.

Q14. Why do we scale features, and which models need it?#

Scaling puts features on comparable ranges — for instance, mean 0 and standard deviation 1. Distance-based and gradient-based models need it: k-nearest neighbours and k-means because a feature measured in thousands drowns one measured in decimals; neural networks and anything trained by gradient descent because wildly different scales make optimisation slow and unstable. Regularised linear models need it because the penalty hits all weights equally, which is only fair if features share a scale.

Tree models do not care — they split on thresholds, and any monotonic rescaling leaves the splits identical. Knowing which models are immune is what distinguishes the answer. And the fitting rule from Q13 applies: fit the scaler on training data only. See feature-engineering.

Q15. Your model was great offline and is failing in production. Walk me through your debugging.#

A checklist question — the interviewer wants ordered hypotheses, not one guess.

1. Data mismatch first: is production input processed identically to training input? Different tokenisation, units, encodings or defaults — this is the most common cause (training-serving skew).
2. Leakage second (Q5): was the offline score ever real? A feature unavailable at prediction time explains everything.
3. Drift third: has the world moved since the training data was collected? Compare production input distributions against training distributions. See monitoring-and-drift.
4. Feedback loops and edge cases last: is the model seeing traffic the training data never covered?

Closing sentence: "and I would want monitoring that catches this before users do" — which is the door to the MLOps conversation.

Common mistakes in this round#

Answering only the direct question. The trade-off sentence is where marks live; volunteer it.

Reciting definitions at maximum speed. One slow, structured answer outscores five rushed ones.

Bluffing at the edge. The viva structure means the interviewer will reach a question you cannot answer. "I do not know, but here is how I would reason about it" scores; confident nonsense ends loops.

Try it yourself#

Cover the answers. Speak each of Q6–Q15 aloud in the four-part shape from the beginner tab, and time yourself — ninety seconds per answer is the target. Then change the degree list in bias_variance.py to (2, 5, 8, 12, 14) and predict the two error columns before running.

What to learn next#

• Statistics questions, with answers — the round that usually shares a slot with this one.
• Overfitting and underfitting — the lesson behind the most-asked question here.
• Model evaluation — precision, recall and the metric questions in full.