When the loss will not go down

Setup#

pip install torch

Captured with torch 2.5.1 on CPU; runs are seeded and deterministic.

A perfectly broken training loop#

This program contains a real bug taken from life. Read it before the output — many reviewers miss it:

import torch
import torch.nn as nn

torch.manual_seed(0)
X = torch.randn(256, 10)
y = (2 * X[:, 0] - X[:, 3] > 0).long()

model = nn.Sequential(nn.Linear(10, 32), nn.ReLU(), nn.Linear(32, 2))
optimizer = torch.optim.Adam(nn.Linear(10, 2).parameters(), lr=1e-3)  # the bug: a different model's parameters
loss_fn = nn.CrossEntropyLoss()

for step in range(300):
    optimizer.zero_grad()
    loss = loss_fn(model(X), y)
    loss.backward()
    optimizer.step()
    if step % 100 == 0:
        print(f"step {step:3d}  loss {loss.item():.4f}")
print(f"ln(2) = {torch.log(torch.tensor(2.0)).item():.4f}  <- the 'predicting nothing' floor for 2 classes")

step   0  loss 0.6936
step 100  loss 0.6936
step 200  loss 0.6936
ln(2) = 0.6931  <- the 'predicting nothing' floor for 2 classes

The optimiser was handed a throwaway model's parameters — a paste error, a renamed variable, a wrapped model whose inner module got passed. Gradients flow beautifully into model; the optimiser updates something else entirely. Nothing crashes, forever.

The two-line experiment that catches it#

Snapshot the weights, take one training step, ask whether anything moved:

before = [p.detach().clone() for p in model.parameters()]
optimizer.zero_grad()
loss = loss_fn(model(X), y)
loss.backward()
optimizer.step()

grads_exist = all(p.grad is not None for p in model.parameters())
weights_moved = any((p != b).any() for p, b in zip(model.parameters(), before))
print(f"gradients computed: {grads_exist}")
print(f"weights changed:    {weights_moved}")

gradients computed: True
weights changed:    False

Gradients yes, movement no — the fault must sit between backward and the weights: the optimiser's parameter list, or a learning rate of zero. That is a two-suspect shortlist from a two-line test.

The magic number in the output#

The stuck loss was 0.6936, and the printed floor 0.6931 — the score for predicting fifty-fifty on every sample. Every class count has such a floor: about 1.10 for 3 classes, about 2.30 for 10. (For the curious, the researcher block derives them.)

Memorise your task's floor. A loss pinned there says: the model outputs the same guess for everything. A loss stuck above it usually means something harsher — a learning rate melting the weights, or labels that do not match the features.

The full checklist, in test order#

1. Weights moving? Run the two-line test. If no: optimiser's parameter list, lr=0, missing optimizer.step().
2. Loss at the guessing floor? Model ignores input: dead ReLUs from a hot learning rate, an all-zero input pipeline, logits fed through a stray softmax.
3. zero_grad present? Missing it makes gradients pile up — loss usually thrashes rather than freezes.
4. Learning rate sane? Sweep it with the range test; both extremes flatline in different styles.
5. Can it memorise 8 samples? The decisive experiment — big enough to deserve its own lesson.

Common mistakes#

Training longer, hoping. A frozen loss at step 300 is a frozen loss at step 30,000. Flat means broken machinery, not insufficient patience.

Changing five things at once. Swap the optimiser, lower the rate, add layers, change the loss — and if it improves you learned nothing about which handbrake it was. One experiment, one variable.

Confusing "not going down" with "noisy". Batch-to-batch loss jitter is normal; judge the trend across hundreds of steps, ideally smoothed, before declaring stuckness.

Blaming the model first. Architecture is the least likely suspect on this list, and the most tempting one. The boring machinery — optimiser wiring, learning rate, data dtypes — earns its place at the top of the checklist by base rate.

Try it yourself#

Fix the bug (model.parameters()), confirm the loss falls, then plant each of these and watch the distinct signatures: lr=0.0, deleting optimizer.step(), and deleting optimizer.zero_grad(). Three broken runs, three recognisably different loss curves.

What to learn next#

• Overfit a single batch first — the decisive experiment, promoted to a ritual.
• Checking that gradients reach every layer — when the two-line test says gradients are the problem.
• Finding a learning rate that works — ruling out the most common single cause.