Loss spikes and gradient clipping

Clipping, formally#

Global-norm clipping (Pascanu et al., 2013, On the difficulty of training recurrent neural networks) rescales the concatenated gradient $g$ of all parameters:

$$ \tilde g = g \cdot \min!\left(1, \frac{\tau}{\lVert g \rVert_2}\right) $$

$\tau$ is the threshold and $\lVert g \rVert_2$ the global L2 norm. The map is the identity inside the ball of radius $\tau$ and a projection onto its boundary outside. Direction is exactly preserved.

This is not a benign modification of gradient descent. Clipped SGD does not converge to a stationary point of the original objective in general; it converges under a relaxed smoothness condition. Zhang et al., 2020 (Why Gradient Clipping Accelerates Training) introduced $(L_0, L_1)$-smoothness, where the local smoothness constant grows with the gradient norm:

$$ \lVert \nabla^2 f(x) \rVert \leq L_0 + L_1 \lVert \nabla f(x) \rVert $$

Under this condition — which they verified empirically on language models — clipped gradient descent has a strictly better convergence rate than any fixed-step-size method. Clipping is not damage control bolted onto SGD; it is the correct algorithm for this loss surface.

What actually causes spikes#

Four mechanisms are documented, and they are distinct.

1. Data × state interaction. PaLM (Chowdhery et al., 2022) saw roughly 20 spikes in the 540B run and none in the 8B or 62B runs at the same data order. The decisive ablation: replaying the offending batches from an earlier checkpoint produced no spike. The conclusion in the paper is explicit — spikes require a specific batch and a specific parameter state. Their fix was operational: rewind ~100 steps, skip 200–500 batches, resume.

2. Adam's update decorrelating from the descent direction. Molybog et al., 2023 (A Theory on Adam Instability in Large-Scale Machine Learning) studied models from 7B to 546B parameters and describe the failure state precisely: Adam enters a regime where the update vector has a large norm and is essentially uncorrelated with the descent direction. The mechanism runs through the ratio $\hat m_t / (\sqrt{\hat v_t} + \epsilon)$. For a parameter whose recent gradients have been near zero, $\hat v_t$ decays until $\epsilon$ dominates the denominator, and the resulting step size no longer reflects the gradient that produced it. They tie the severity to large batch sizes, which is exactly the regime of LLM pretraining. The practical levers this implicates are $\epsilon$, $\beta_2$, and batch size — all of which are tuned in practice, and none of which is a complete fix.

3. Attention-logit growth. Wortsman et al., 2023 (Small-scale proxies for large-scale Transformer training instabilities) reproduced large-model instabilities at small scale by raising the learning rate, then showed two fixes generalise: qk-layernorm (normalise Q and K before the dot product) removes attention-logit divergence, and a z-loss $\lambda \log^2 Z$ on the output softmax normaliser $Z$ removes output-logit divergence. PaLM used $\lambda = 10^{-4}$.

4. Embedding-layer gradient scale. Takase et al., 2024 (Spike No More) argue spikes originate in large embedding gradients and derive initialisation conditions bounding the update norm. GLM-130B (Zeng et al., 2023) shipped embedding gradient shrink, scaling the embedding gradient by $\alpha \approx 0.1$ via a straight-through trick, for the same reason.

Adaptive alternatives to a fixed threshold#

A fixed $\tau$ has a defect: as training proceeds the healthy gradient norm falls, so a threshold set at step 1000 becomes progressively less protective.

• AutoClip (Seetharaman et al., 2020) sets $\tau$ to a running percentile of observed norms.
• ZClip (2025) models the norm's history with an exponential moving average and standard deviation, clipping on a z-score rather than an absolute value. Reported to reduce spike frequency in LLM pretraining relative to fixed clipping.

Both share a failure mode worth knowing: if the norm rises slowly, an adaptive threshold rises with it and stops protecting anything.

The monitoring signal#

Log $\lVert g \rVert_2$ every step, and watch its distribution rather than the loss. In a healthy run, $\log \lVert g \rVert$ is close to stationary with a slow downward trend and light tails. The observable precursors of a spike, in order of how early they appear:

1. Rising maximum attention logit.
2. Rising output-logit magnitude (equivalently, falling output entropy).
3. Gradient-norm outliers.
4. The loss itself.

By the time (4) is visible on a dashboard the step has already been applied. This is why the pre-clip norm belongs in your logging, and why serious runs also log per-layer norms — a spike localised to one layer points at a very different cause from a global one.

Papers#

• Pascanu et al., On the difficulty of training recurrent neural networks, 2013 — arxiv.org/abs/1211.5063
• Zhang et al., Why Gradient Clipping Accelerates Training, ICLR 2020 — arxiv.org/abs/1905.11881
• Seetharaman et al., AutoClip, 2020 — arxiv.org/abs/2007.14469
• Chowdhery et al., PaLM, 2022 — arxiv.org/abs/2204.02311 (section 5.1 documents the spikes)
• Zeng et al., GLM-130B, 2023 — arxiv.org/abs/2210.02414
• Molybog et al., A Theory on Adam Instability in Large-Scale Machine Learning, 2023 — arxiv.org/abs/2304.09871
• Wortsman et al., Small-scale proxies for large-scale Transformer training instabilities, 2023 — arxiv.org/abs/2309.14322
• Takase et al., Spike No More, 2024 — arxiv.org/abs/2312.16903
• Kumar et al., ZClip: Adaptive Spike Mitigation for LLM Pre-Training, 2025 — arxiv.org/abs/2504.02507

What to learn next#

• Reading a training run while it happens — the dashboard that catches this early.
• Debugging NaN loss — the version of this problem that kills the run outright.
• Mixed precision training — why bf16 removes a whole class of spikes.