Learn AI Create the Future — with Pranay
Main site

Statistics and Experiments Section 016

Experiment Design and A/B Testing

Running an online experiment that gives you an answer you can defend, and spotting the ways it goes wrong.

Every lesson in this section is written by .

10 of 10 lessons published Three reading levels on every lesson

Start with “Randomisation units and assignment”

Lessons in order

Work top to bottom. Each lesson assumes the one above it.

  1. Randomisation units and assignment
  2. A/A tests
  3. Peeking and sequential testing
  4. Sample ratio mismatch
  5. CUPED and variance reduction
  6. Novelty and primacy effects
  7. Interference and network effects
  8. Switchback experiments
  9. Guardrail metrics
  10. Bayesian A/B testing