Too small a model, or trained too little?

Separating approximation, optimisation and estimation error#

The excess risk of a trained model decomposes into three terms, and this lesson is a procedure for attributing an observed gap to the right one. With $\mathcal{H}$ the hypothesis class, $h^\star$ the best possible predictor, $h^\star_{\mathcal{H}}$ the best member of $\mathcal{H}$, $\hat{h}$ the empirical risk minimiser and $\tilde{h}$ what the optimiser actually returned after a finite budget,

$$ R(\tilde h) - R(h^\star) = \underbrace{R(h^\star_{\mathcal{H}}) - R(h^\star)}{\text{approximation}} + \underbrace{R(\hat h) - R(h^\star{\mathcal{H}})}{\text{estimation}} + \underbrace{R(\tilde h) - R(\hat h)}{\text{optimisation}} . $$

"Model too small" is approximation error; "trained too little" is optimisation error; "memorising" is estimation error. The training loss isolates the first two from the third, because it contains no held-out sample, which is why the diagnostic column of the table is a training-loss slope rather than anything measured on validation data. Bottou and Bousquet (2008), The Tradeoffs of Large Scale Learning (NeurIPS), formalise the budgeted version of this decomposition and derive the counter-intuitive practical result that under a fixed compute budget the optimal choice is often a worse optimiser run on more data — the same reason the largest model with the shortest schedule wins on this page.

Why two hidden units genuinely cannot#

Universal approximation (Cybenko, 1989; Hornik, 1991) guarantees arbitrary accuracy only as width grows without bound; it says nothing at width 2. For a ReLU network the function class is piecewise linear, and the number of linear regions is bounded above by a quantity that grows polynomially in width and exponentially in depth — Montúfar et al. (2014), On the Number of Linear Regions of Deep Neural Networks (NeurIPS). A target with an $x_0x_1$ interaction requires a decision boundary that a two-region-per-layer network cannot bend into shape, so the approximation term is bounded away from zero for reasons that have nothing to do with the optimiser.

This also explains the feature-engineering exercise: supplying $x_0x_1$ explicitly changes $\mathcal{H}$, collapsing the approximation term to near zero at unchanged parameter count. Capacity is a property of the model and its input representation, which is why "add parameters" and "add features" are alternative treatments for one diagnosis.

Compute allocation between size and steps#

The trade-off this lesson explores by hand is the central question of large-scale training. Kaplan et al. (2020), Scaling Laws for Neural Language Models, fit loss as a joint power law in parameters, data and compute and concluded that under a fixed compute budget one should train very large models for relatively few steps and stop well short of convergence — a direct formalisation of "hidden=64, epochs=100". Hoffmann et al. (2022) (Chinchilla) revised the exponents, finding parameters and tokens should scale roughly in proportion, and demonstrated a 70B model trained on 1.4T tokens beating a 280B model trained on far fewer. The methodological lesson beneath both is the one in the walkthrough: the optimum lives on a two-dimensional surface, and a one-dimensional sweep can miss it entirely.

Practically, this argues for an IsoFLOP design — hold a compute budget fixed, vary the size/steps split along it, and repeat at two or three budgets — rather than a full grid. The envelope of those curves is what extrapolates; individual points do not.

The overfit-a-tiny-subset test, formally#

Training to near-zero loss on $n = 50$ examples asks whether $\mathcal{H}$ can shatter a small sample under the actual optimiser. Zhang et al. (2017), Understanding Deep Learning Requires Rethinking Generalization (ICLR), established the empirical fact that standard architectures can fit random labels on full-size datasets, which makes failure on fifty real examples strongly diagnostic: it implies a defect in capacity, conditioning or wiring rather than an intrinsic difficulty of the task. The test is cheap, has near-zero false-positive rate in practice, and is the reason it belongs before every long run.

Its limitation is worth stating. Passing rules out gross capacity failure but says nothing about the sample complexity of the real task, so it can leave a genuine underfit undetected when the target requires many more effective parameters than fifty points can reveal. The complementary check is the shape of the training loss against training-set size — the learning curve — where a training loss that stays high as $n$ grows is the population-level signature of the same ceiling.

What to learn next#

• Finding a learning rate — the cause that impersonates a capacity ceiling most often.
• Learning curves — the same question asked against data size rather than training time.
• Deciding a model is good enough — what to do once neither cause applies any more.