Train, test and validation splits

What the split is estimating#

Splitting is estimation of the risk functional:

R(h) = E_{(x,y) ~ D} [ L( h(x), y ) ]

A held-out estimate R_hat is unbiased for R(h) only when the held-out sample is independent of everything used to produce h. Every failure mode in this lesson is a violation of that independence, not a violation of the i.i.d. assumption itself.

Two distinct quantities are routinely conflated:

• Conditional risk R(h_S) — performance of the specific model fitted on this training set S
• Expected risk E_S[ R(h_S) ] — performance of the procedure, averaged over training sets

A single hold-out estimates the first. k-fold cross-validation estimates something closer to the second, and Bengio & Grandvalet (2004) proved there is no unbiased estimator of the variance of k-fold CV, because the folds share training data. This is why cross-validated t-tests are invalid without the correction of Nadeau & Bengio (2003).

Split ratio#

The trade-off is between bias in the fitted model (small training set) and variance in the estimate (small test set). For accuracy a measured on n_test points:

SE = sqrt( a * (1 - a) / n_test )

At a = 0.9, n_test = 200 gives SE ≈ 0.021, so a 95 percent interval spans roughly 0.86 to 0.94. Detecting a 1-point difference requires thousands of test points, which is why leaderboard margins on small benchmarks are usually noise. Use the Wilson interval rather than the normal approximation for small n or extreme rates.

Kearns (1996) analysed the ratio theoretically; the practical answer is that 20–30 percent is customary for moderate n, and that for large n the test set should be sized by the precision you need rather than by a percentage. Ten million rows do not need two million held out.

A taxonomy of leakage#

Kaufman et al. (2012) give the standard treatment, distinguishing:

• Leakage in training examples — information from the target crossing into features, e.g. a column populated after the outcome
• Leakage in feature construction — statistics computed over the full dataset before splitting
• Leaking rows — duplicates or group members spanning the split

Kapoor & Narayanan (2023) surveyed 294 papers across 17 scientific fields and found leakage-driven irreproducibility to be pervasive. They propose model info sheets as a reporting standard. Their taxonomy adds illegitimate feature leakage and temporal leakage as separate categories, and documents entire subfields whose reported gains vanish under corrected evaluation.

The selection-before-splitting experiment in the Developer section reproduces the result of Ambroise & McLachlan (2002), who showed that gene-selection-before-cross-validation produced near-zero apparent error rates on datasets with no signal. Hastie, Tibshirani & Friedman (2009), section 7.10.2, present the same demonstration as the canonical wrong way to cross-validate.

Structured splits#

Grouped. Required whenever rows share a latent unit. GroupKFold, LeaveOneGroupOut, StratifiedGroupKFold. Choosing the grouping variable is a modelling decision that encodes the deployment question: splitting by patient asks "will this work on a new patient", splitting by hospital asks "will this work at a new hospital", and those are different, increasingly demanding questions.

Temporal. Beyond a forward-chaining split, financial machine learning requires more care. López de Prado (2018) introduces purging — removing training observations whose label horizons overlap the test period — and embargo, dropping a buffer immediately after the test window to break serial correlation. Without these, overlapping labels leak across the boundary even when the split respects time order.

Spatial. Spatial autocorrelation makes neighbouring points near-duplicates. Roberts et al. (2017) recommend blocked cross-validation with blocks larger than the autocorrelation range.

Chemical. Random splits of molecular datasets are optimistic because analogues appear on both sides. Scaffold splits (Wu et al., 2018, MoleculeNet) group by Bemis-Murcko scaffold and are the accepted standard.

Adaptive data analysis#

The reason the test pile must be opened once is formalisable, and this is the most interesting theory in the area.

Classical guarantees assume the hypothesis is fixed before the holdout is seen. Once analysts choose what to try based on holdout results, the analysis is adaptive, and after k adaptive rounds the holdout can be overfitted at a rate scaling roughly as sqrt(k / n).

Dwork et al. (2015) showed that differential privacy provides a guarantee against this. Adding calibrated noise to reported holdout statistics — the reusable holdout — supports exponentially many adaptive queries while preserving validity. The Ladder mechanism (Blum & Hardt, 2015) applies the same idea to competition leaderboards, releasing a score only when it improves on the best so far by a margin.

Empirically, adaptive overfitting is milder than the theory permits. Roelofs et al. (2019) analysed Kaggle competitions and found little evidence of substantial leaderboard overfitting in practice, likely because submissions are highly correlated rather than adversarially chosen.

Does a test set generalise?#

Recht et al. (2019) constructed new test sets for CIFAR-10 and ImageNet by replicating the original collection process as closely as possible. Every model dropped — 3 to 15 percent on CIFAR-10, 11 to 14 percent on ImageNet.

The crucial finding is that accuracy on the new sets was almost perfectly linearly related to accuracy on the old ones. So a decade of progress was not leaderboard overfitting; the drop reflects distribution shift from irreproducible details of data collection. Relative model ordering survived. Absolute numbers did not.

Gorman & Bedrick (2019) made a sharper point for NLP: replacing standard splits with random splits changes system rankings in a majority of the POS-tagging systems they examined. Conclusions drawn from a single canonical split may not survive resplitting.

Nested cross-validation#

When hyperparameters are tuned, the tuning is part of the procedure and must sit inside the evaluation loop:

outer loop (k folds)   ->  estimates generalisation
    inner loop (m folds) ->  selects hyperparameters on the outer training fold only

Cost is k * m fits. Cawley & Talbot (2010) document how widespread the shortcut of reporting the inner-loop score is, and how large the resulting optimistic bias can be — often exceeding the differences between the methods being compared. See overfitting and underfitting for the estimator comparison table.

sklearn.model_selection.cross_val_score(GridSearchCV(...), ...) implements this correctly in one line. Note that nested CV estimates the risk of the procedure, not of any single fitted model; the final model is refitted on everything afterwards.

Key references#

• Kearns, M. (1996). A Bound on the Error of Cross Validation Using the Approximation and Estimation Rates. Neural Computation 9(5).
• Ambroise, C. & McLachlan, G. (2002). Selection Bias in Gene Extraction on the Basis of Microarray Gene-Expression Data. PNAS 99(10).
• Nadeau, C. & Bengio, Y. (2003). Inference for the Generalization Error. Machine Learning 52(3).
• Bengio, Y. & Grandvalet, Y. (2004). No Unbiased Estimator of the Variance of K-Fold Cross-Validation. JMLR 5.
• Cawley, G. & Talbot, N. (2010). On Over-fitting in Model Selection and Subsequent Selection Bias in Performance Evaluation. JMLR 11.
• Kaufman, S., Rosset, S. & Perlich, C. (2012). Leakage in Data Mining. ACM TKDD 6(4).
• Dwork, C. et al. (2015). The Reusable Holdout: Preserving Validity in Adaptive Data Analysis. Science 349(6248).
• Roberts, D. et al. (2017). Cross-validation Strategies for Data with Temporal, Spatial, Hierarchical, or Phylogenetic Structure. Ecography 40(8).
• López de Prado, M. (2018). Advances in Financial Machine Learning. Wiley. Chapter 7 on purging and embargo.
• Recht, B. et al. (2019). Do ImageNet Classifiers Generalize to ImageNet? ICML.
• Gorman, K. & Bedrick, S. (2019). We Need to Talk about Standard Splits. ACL.
• Roberts, M. et al. (2021). Common Pitfalls and Recommendations for Using Machine Learning to Detect and Prognosticate for COVID-19 Using Chest Radiographs and CT Scans. Nature Machine Intelligence 3.
• Kapoor, S. & Narayanan, A. (2023). Leakage and the Reproducibility Crisis in Machine-Learning-Based Science. Patterns 4(9).

Current state#

Evaluation design is now understood to be a larger source of error in published results than model design. Roberts et al. (2021) reviewed 62 COVID-19 imaging models from over 2,200 papers and found none clinically usable, citing duplicated data across public sources, non-grouped splits, and unspecified provenance.

Three shifts follow from this literature.

Multiple splits, not one. Report variability across resplits rather than a single number, given Gorman & Bedrick.

Splits designed around the deployment question. The split should be the hardest one your deployment requires — new patients, new sites, next quarter — not the easiest one your data permits.

Evaluation as an artefact to review. Model info sheets (Kapoor & Narayanan), model cards (Mitchell et al., 2019), and datasheets (Gebru et al., 2021) all push evaluation protocol into the reviewable record. The trend is towards treating the split as a claim requiring justification, rather than a preprocessing step.

What to learn next#

• Model evaluation — which numbers to compute once you have an honest split.
• Overfitting and underfitting — the gap between the piles, and what it means.
• Feature engineering — the work that must happen inside the pipeline, not before it.