Learn AI Create the Future — with Pranay
Main site

AI glossary

Gradient clipping

In one sentence Gradient clipping caps the size of each update during training, so one extreme batch cannot blow the model up.

By Updated

Gradient clipping puts a ceiling on how large a gradient is allowed to be, so a single wild batch cannot wreck the weights.

It is the speed governor fitted to a school bus. The bus drives normally almost all the time. But when the driver floors it — deliberately or in panic — the governor caps the speed at a safe limit. Gradient clipping does the same for training: normal updates pass through untouched, and rare huge ones get scaled down to a set maximum.

Why do huge gradients happen? An unusual batch, an unlucky spot on the loss surface, or the compounding effect of backpropagation through many steps — the exploding cousin of the vanishing-gradient problem, common in RNNs and deep transformers. One such spike can throw the weights somewhere terrible, and the loss curve shows it as a sudden cliff-jump upward or a NaN.

The standard form is clipping by global norm: measure the combined length of all gradients, and if it exceeds a threshold (often 1.0), scale everything down proportionally so direction is preserved.

python
torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0)

It is cheap insurance, one line in most training loops, and standard practice in RNN and LLM training. It treats the symptom, though — gradients that spike constantly usually point at a learning-rate set too high or unstable data.

Where to go next