Statistics questions, with answers

Setup#

Questions 6–15, with runnable code where code answers best.

pip install numpy

Runs on CPU in about a second. Outputs made with NumPy 2.4; identical seeds reproduce them on any recent version.

Q6. Design and analyse an A/B test for me#

The design half: randomise users into two groups, decide the metric and the sample size before starting, run to the planned size, then test. The analysis half is beautifully demonstrated by the permutation test — an approach worth knowing because it also explains what a p-value is:

import numpy as np

rng = np.random.default_rng(0)

# minutes watched per user in a tiny A/B test - invented numbers, realistic scale
a = np.array([12.1, 8.4, 15.2, 9.8, 11.5, 7.9, 13.3, 10.6, 9.1, 12.7])   # old design
b = np.array([14.9, 11.2, 16.8, 10.4, 13.1, 12.6, 15.5, 9.9, 14.2, 13.8])  # new design

observed = b.mean() - a.mean()
pooled = np.concatenate([a, b])

# if the design changed nothing, the group labels are meaningless -
# so shuffle the labels many times and see how often chance beats reality
extreme = 0
n_shuffles = 100_000
for _ in range(n_shuffles):
    rng.shuffle(pooled)
    diff = pooled[10:].mean() - pooled[:10].mean()
    if abs(diff) >= abs(observed):
        extreme += 1

print(f"observed difference:  {observed:.2f} minutes")
print(f"permutation p-value:  {extreme / n_shuffles:.4f}")

# bootstrap: resample each group with replacement to see how much the
# difference itself wobbles from sample to sample
diffs = np.array([
    rng.choice(b, size=10).mean() - rng.choice(a, size=10).mean()
    for _ in range(10_000)
])
lo, hi = np.percentile(diffs, [2.5, 97.5])
print(f"95% bootstrap interval for the difference: [{lo:.2f}, {hi:.2f}] minutes")

observed difference:  2.18 minutes
permutation p-value:  0.0501
95% bootstrap interval for the difference: [0.28, 4.06] minutes

That p-value landed at 0.0501 — a fraction above the traditional 0.05 line — and that is a gift of an interview moment. The weak candidate says "not significant, discard". The strong candidate says: 0.0501 and 0.0499 describe the same evidence; the difference is 2.2 minutes with an interval excluding zero; with ten users per group the right conclusion is "promising, underpowered — run it bigger". Treating 0.05 as a cliff edge is exactly the misunderstanding this round probes.

Q7. What is the bootstrap, and when would you reach for it?#

The bootstrap re-samples your data with replacement many times, recomputing your statistic each time, and reads the spread of results as the uncertainty of that statistic. It is the second half of the code above, and it answers "how much would this number change if I could rerun the experiment?" — without rerunning anything.

Reach for it when no textbook formula exists for your statistic's uncertainty: medians, ratios, a model's AUC, revenue per user with its wild skew. Limitations to volunteer: it struggles with very small samples (resampling ten points only rearranges those ten values) and with extremes like the maximum.

Q8. A disease test is 99% accurate. You test positive. How worried should you be?#

The most famous statistics interview question, and the answer shocks everyone once. Suppose 1 person in 1,000 has the disease, the test catches 99% of sick people, and it false-alarms on 5% of healthy people.

Take 100,000 people. About 100 are sick, and the test catches 99 of them. About 99,900 are healthy, and 5% false-alarm — 4,995 people. Your positive result puts you in a room of 99 + 4,995 = 5,094 positives, of whom 99 are sick:

 P(sick | positive) = 99 / 5,094 ≈ 1.9%

Under 2%, because the disease is rare and false alarms swamp true ones. This is Bayes' theorem in action — updating a belief with evidence, where the base rate (how common the condition is) dominates. The interview transfer: a 99%-accurate fraud model drowning in false positives on rare fraud is the same arithmetic, which connects straight to precision and recall in ml-theory-questions. Foundations: probability.

Q9. What is statistical power, and what increases it?#

Power is the probability your test detects an effect that genuinely exists — the complement of the Type II error rate. Underpowered tests are how real improvements get killed: the effect was there, the test could not see it, the feature died.

Four levers: bigger samples, bigger true effects (test bold changes, not button shades), less noisy metrics, and a looser significance threshold. The practice sentence: decide power before the test, by computing the sample size needed to detect the smallest effect worth shipping — never after.

Q10. Why can't I check my A/B test daily and stop when it becomes significant?#

Because you are giving luck many chances and keeping its best attempt. Each daily peek is another lottery ticket; across weeks of peeking, the chance of at least one accidental "significant!" balloons far past 5%. The test's guarantee applied to one look at a pre-planned size.

Name the practice: this is the peeking problem, and it produces winners that evaporate after shipping. The fixes: commit to the sample size in advance, or use methods built for continuous monitoring — sequential tests, which spend an error budget across looks. Knowing that such methods exist is the senior part of the answer.

Q11. You measured 20 metrics and one moved significantly. What do you tell your boss?#

That this is expected under pure chance: at a 5% threshold, one-in-twenty metrics false-alarms on average even when nothing changed. Finding exactly one mover in twenty is what "no effect anywhere" looks like.

The multiple comparisons problem, by name. Remedies: pick one primary metric before the experiment and let the rest be exploratory; or correct the threshold — Bonferroni divides it by the number of tests (crude, safe), false-discovery-rate methods are gentler at scale. The one-liner that lands: "if you torture the data long enough, it will confess to anything."

Q12. Standard deviation versus standard error — what is the difference?#

Standard deviation measures how much individual data points spread around their mean — a fact about the population that more data does not shrink. Standard error measures how much the sample mean itself would wobble across repeated samples — and it shrinks as the sample grows, falling with the square root of the sample size.

The square-root part is worth saying: four times the data halves the standard error. Precision gets expensive fast — a tenth of the error costs a hundred times the sample. Interviewers probe this with: "your mean is stable but users vary hugely — contradiction?" No: small standard error, large standard deviation, at the same time.

Q13. What is Simpson's paradox? Construct an example#

A trend that holds in every subgroup can reverse when the subgroups are pooled. The canonical real dataset — kidney-stone treatments, Charig et al. (1986):

A wins on small stones. A wins on large stones. Pooled, B "wins". The reversal happens because A — the more invasive treatment — was given mostly the hard cases, so its pooled rate is dragged down by case mix, not by quality.

The lesson to state: pooled comparisons silently assume the groups faced the same mix of cases. When a lurking variable decides who gets what, subgroup analysis — or better, randomisation — is the defence.

Q14. When is the mean the wrong summary?#

Whenever the distribution is skewed or heavy-tailed, because a few extreme values drag the mean away from the typical case. Nine people earning 30,000 rupees and one earning 3,000,000 have a mean income of 327,000 — describing nobody. The median (50th percentile) says 30,000 and describes almost everyone.

The engineering version scores extra: latency. Mean response time hides disasters; the 95th and 99th percentiles are what users feel. "Report medians and tail percentiles for anything skewed — money, latency, session length" is the practitioner's answer.

Q15. What is regression to the mean? Why do managers believe punishment works?#

Extreme results are part skill, part luck, and the luck does not repeat. The best performer this month was both good and lucky; next month the luck resets and they look worse — with no real change. The worst performer improves the same way.

The famous consequence (Kahneman's flight instructors): punish after terrible performance, and things improve anyway, because extremes drift back towards normal. Praise after brilliance, and things decline anyway. Punishment gets false credit; praise gets false blame. The interview transfer: "our worst-performing stores improved after the intervention" is uninterpretable without a control group, because the worst stores were partly unlucky and would have rebounded regardless.

Common mistakes in this round#

Reciting mechanics without the trap. Every question above has a famous misreading; naming it is most of the score.

Speaking in formulas. This round is spoken. The candidate who can explain a p-value to a smart non-statistician outscores the one who writes integrals.

Forgetting the business. "Significant" and "worth shipping" are different claims — a real effect can be too small to matter. End answers with the decision, not the statistic.

Try it yourself#

Rerun permutation_ab.py after changing one user's number in group b — make 9.9 into 13.9 — and predict the p-value's direction before running. Then answer Q8 aloud with a 1-in-100 base rate instead of 1-in-1,000, doing the people-count arithmetic as you speak.

What to learn next#

• Deep learning questions, with answers — the next question set in the loop.
• Statistics — the foundations behind every answer here.
• Probability — Bayes' theorem and the disease-test arithmetic, taught fully.