Plateaus, sudden drops and double descent

Plateaus: saddles, symmetry and the escape time#

Flat regions in deep networks are dominated by saddle points rather than local minima. Dauphin et al. (2014), Identifying and Attacking the Saddle Point Problem in High-dimensional Non-convex Optimization (NeurIPS), argue from random matrix theory that in high dimension critical points with small loss are overwhelmingly saddles: for a random Hessian, the probability that all $n$ eigenvalues are positive falls exponentially in $n$, and the index of a critical point concentrates as a function of its loss. Gradient descent near a strict saddle escapes along the negative-curvature direction, but the escape time scales with the inverse of the negative eigenvalue's magnitude — arbitrarily long plateaus for arbitrarily flat saddles.

A second, distinct mechanism is permutation and sign symmetry. At initialisation, hidden units of a layer are exchangeable; the loss surface contains manifolds where units are functionally identical, and the network cannot make progress until symmetry breaks. Saad and Solla (1995), On-line Learning in Soft Committee Machines (Physical Review E), derived exactly this for the teacher-student setting: a long symmetric plateau in which all student units align to the same teacher direction, ending in a specialisation transition. Parity is a canonical hard case — its Fourier spectrum is concentrated on a single degree-$k$ character, so every subset of fewer than $k$ inputs is uninformative, and the statistical-query lower bounds of Kearns (1998) imply that gradient-based learners need either many queries or high precision.

Grokking (Power et al., 2022, Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets) is the sharpest documented version: training accuracy saturates early while validation stays at chance for orders of magnitude more steps, then jumps. Subsequent analyses (Nanda et al., 2023, Progress Measures for Grokking via Mechanistic Interpretability, ICLR) show the transition is not sudden internally — a Fourier-basis circuit forms gradually and the metric is what changes discontinuously. The practical lesson generalises well beyond modular arithmetic: loss is a lagging indicator of internal structure, and progress measures computed from the weights can reveal movement during an apparently dead plateau.

Double descent: the interpolation threshold#

Belkin et al. (2019), Reconciling Modern Machine Learning Practice and the Classical Bias–Variance Trade-off (PNAS), named the shape and located the peak at the interpolation threshold, where model capacity first suffices to fit the training set exactly. Nakkiran et al. (2021), Deep Double Descent (ICLR 2020; JSTAT 2021), demonstrated it in modern networks and identified three axes along which it appears — model size, training epochs, and dataset size (where more data can hurt at fixed model size) — unified by an effective model complexity.

The linear case is solved. For minimum-norm least squares with $p$ features and $n$ samples, the risk diverges as $p \to n$ because the smallest singular value of the design matrix approaches zero, so $|\hat\beta| = |F^{+}y|$ blows up; the Marchenko–Pastur law gives the precise rate. Hastie, Montanari, Rosset and Tibshirani (2022), Surprises in High-Dimensional Ridgeless Least Squares Interpolation (Annals of Statistics), give exact asymptotic risk in the proportional regime $p/n \to \gamma$, and their central practical result is that optimally-tuned ridge regression removes the peak entirely and is monotone in $\gamma$. Double descent is therefore best understood as a pathology of implicit regularisation being insufficient near the threshold, not as evidence against the bias–variance decomposition. Belkin, Rakhlin and Tsybakov (2019) and Bartlett et al. (2020) (Benign Overfitting in Linear Regression, PNAS) characterise when interpolation of noise is harmless: the spectrum of the covariance must have enough small directions to absorb the noise without corrupting the signal directions.

What this changes about reading a curve#

Three revisions to the default rules, each narrow.

• "Flat means dead" becomes "flat means measure the gradient". Gradient norm, weight-norm drift and a task-specific progress measure separate a live plateau from a dead one at negligible cost.
• "Validation rising means stop" keeps its default status. Epoch-wise double descent requires label noise and particular capacity regimes; treating it as a routine possibility would justify burning compute on genuinely failed runs. Keep early stopping, and note the exception when the setting matches (when validation loss rises but accuracy improves is the far more common cause of that shape).
• "Bigger got worse, so stop scaling" becomes a sampling question. Two adjacent sizes cannot distinguish a real ceiling from a threshold peak. Sweep capacity geometrically, and tune regularisation at each size, since an untuned penalty is what makes the peak visible in the first place.

What to learn next#

• Too small a model, or trained too little? — separating the two causes this lesson deliberately mixed.
• Overfit one batch — the fastest test for whether a plateau is a bug.
• Buying accuracy with size, and when to stop — how to sweep capacity without wasting a month.