Doing the Work Section 022

Reproducibility and Running Experiments

Getting the same number twice, and keeping track of hundreds of runs without losing your mind.

6 of 6 lessons published Three reading levels on every lesson

Start with “Report a range, not a single number”

Lessons in order

Work top to bottom. Each lesson assumes the one above it.

  1. Report a range, not a single number
  2. Configs instead of constants scattered everywhere
  3. Making sense of two hundred runs
  4. When you cannot reproduce your own number
  5. Designing an ablation that proves your claim
  6. Is this improvement real?