When your validation curve is too noisy to trust

Two variance components, and only one of them is fixable by data#

Write the measured validation score of a run as

$$ \hat{M} = M(\theta(\omega)) + \varepsilon(V) $$

where $\omega$ collects the training randomness (initialisation, batch ordering, dropout masks, augmentation draws, non-deterministic kernel reductions), $\theta(\omega)$ is the resulting parameter vector, $M$ is the population metric, and $\varepsilon(V)$ is the sampling error of evaluating on the finite set $V$. The total variance decomposes as

$$ \operatorname{Var}(\hat{M}) = \underbrace{\operatorname{Var}\omega!\big(M(\theta(\omega))\big)}{\text{training variance}} + \underbrace{\mathbb{E}\omega\big[\operatorname{Var}(\varepsilon \mid \theta)\big]}{\text{evaluation variance}} . $$

Enlarging $V$ attacks only the second term, at rate $1/|V|$. For accuracy, $\varepsilon$ is binomial, so the standard error is $\sqrt{p(1-p)/|V|}$ — the source of the "±7 points at $n=100$" figure, since $1.96\sqrt{0.85 \cdot 0.15/100} \approx 0.070$. Averaging over seeds attacks only the first term, at rate $1/k$ for $k$ seeds. Neither operation substitutes for the other, and a paper reporting five seeds on a 200-row validation set has controlled the smaller of its two variances.

Bouthillier et al. (2021), Accounting for Variance in Machine Learning Benchmarks (MLSys), measure both components empirically across standard benchmarks and reach a counter-intuitive practical conclusion: randomising all sources of variation across a modest number of runs estimates the performance distribution better than fixing seeds and running more of the same configuration, and many published gains lie inside the resulting distribution. Reimers and Gurevych (2017), Reporting Score Distributions Makes a Difference (EMNLP), showed the same for sequence labelling — score differences reported as improvements were frequently within seed spread.

The selection bias in "best epoch"#

Choosing $\hat{t} = \arg\min_t \hat{R}_V(\theta_t)$ and then reporting $\hat{R}V(\theta{\hat{t}})$ reports the minimum of a correlated noisy sequence, which is downward-biased. For $T$ evaluations with per-evaluation noise standard deviation $\sigma$ and weak correlation, the expected optimism grows roughly like $\sigma\sqrt{2\ln T}$ — increasing evaluation frequency inflates the reported score without improving the model. This is the same winner's curse that governs test-set overfitting and hyperparameter search, and it is why an unbiased estimate requires a set never used for selection.

Two mitigations with different costs. A three-way split (train / validation for selection / test for reporting) is clean and expensive in data. Nested cross-validation is data-efficient and expensive in compute; Varma and Simon (2006) (BMC Bioinformatics) quantify the bias of the non-nested alternative on small samples, where it is large enough to invert conclusions.

Deciding whether a difference is real#

The correct test is paired: evaluate both models on the same validation examples and analyse the per-example differences, which removes the shared difficulty of the examples from the variance. McNemar's test is the standard choice for paired binary correctness; Dietterich (1998), Approximate Statistical Tests for Comparing Supervised Classification Learning Algorithms (Neural Computation), evaluates five candidate tests for Type I error and recommends McNemar or 5×2-fold cross-validated $t$ over the resampled $t$-test, which is badly miscalibrated because its folds are not independent. The bootstrap over validation examples gives a confidence interval on the difference without distributional assumptions and generalises to metrics with no closed-form variance (bootstrapping).

For curves specifically, the object of inference is a whole trajectory, and pointwise intervals under-cover when read across many epochs. Reporting a band from independent seeds and refusing to interpret crossings inside it is the cheap, defensible practice; statistical significance in ML and comparing models fairly carry the full apparatus.

Non-determinism you did not choose#

Even with every seed fixed, exact reproduction can fail. Atomic accumulation order on GPU, cuDNN algorithm selection, TF32 and mixed-precision rounding, and multi-worker data loading all introduce run-to-run differences (random seeds and reproducibility). Forcing deterministic kernels costs throughput and does not remove the scientific problem — a result that survives only under one bit-exact configuration was never robust. Determinism is a debugging tool; seed spread is the scientific report.

What to learn next#

• Seed variance and error bars — how to report the spread you have now measured.
• How smoothing and log scales mislead you — the wrong way to make a noisy curve look calm.
• K-fold cross-validation — the fix when there is not enough data for a big validation set.