How to read a loss curve
Four shapes cover almost every loss curve you will ever plot — healthy, exploding, bouncing and overfitting — and telling them apart takes one glance once you know what each looks like.
- 13 min read
- 3 reading levels
- Published
Read these first
On this page 5
One lesson, three depths. Pick the one that fits you today — you can switch any time.
Beginner — No maths. Plain English.
A loss curve charts how wrong your model is, round by round — and its shape names the problem.
Think of the temperature chart clipped to the foot of a hospital bed. The chart cures nothing. It tells the doctor which illness to treat. A steady fall means recovery, and a spike at 3 a.m. means something happened at 3 a.m.
Your loss curve is that chart. Loss is one number for how wrong the model's answers are, and smaller is better. Plot it after every round of training and you get a story, not a snapshot.
Reading that story is the difference between fixing a model in ten minutes and tuning blindly for a week.
Why this had to be invented
Training hands you one number at the end: the final score. A bad final score has many possible causes, and the number names none of them.
Did the model learn too slowly? Did it memorise the practice questions? Did the whole run break in the first minute and then spend six hours doing nothing? A single score cannot say. The history can.
So people started drawing two lines instead of one. One line is the loss on the data the model is learning from, called the training loss. The other is the loss on data the model has never been shown, called the validation loss.
The gap between those two lines, and the direction each one travels, is where the diagnosis lives.
How it works
Four shapes cover most of what you will ever see.
1. HEALTHY 2. EXPLODING 3. BOUNCING 4. OVERFITTING
loss loss loss loss
\ / \ /\ /\ \
\ train / \/ \/ \ \ train
\___ / \____
\___ val / (then it dies) both lines jerk ___/ val
------ / up and down / (rising!)
both fall, shoots upward, no steady fall, train falls,
stay close often to "nan" wobbles forever val turns upward
fix: nothing fix: smaller steps fix: smaller steps fix: stop earlierShape 2 and shape 3 are the same illness at different strengths. The model takes steps that are too big while learning, so it overshoots the answer instead of settling on it. That step size is called the learning rate.
Shape 4 is a different illness. The model has started memorising its practice set instead of learning the pattern — the student who learnt last year's question paper by heart.
This part confuses almost everyone the first time. Read it twice; that is normal. A rising validation line with a falling training line means memorising. Both lines rising means a broken learning rate. Same upward direction, completely different cure.
A real example you have seen
Open any fitness app that tracks weight. The daily dots jump around, because water, dinner and the time of day all move the number. The trend line still slopes down, month over month.
Nobody panics over one heavy Tuesday. They ask whether the trend is falling. Loss curves get read the same way, and for the same reason — every single point carries noise.
Remember this
- Always plot two lines: loss on training data, and loss on held-out data.
- Both falling and close together is healthy. Nothing to fix.
- Both rising or jumping means steps too big. Training down while validation climbs means memorising.
What to learn next
- When validation loss rises but accuracy improves — the first curve that will contradict this lesson.
- Finding a learning rate — how to avoid shapes 2 and 3 before you ever plot them.
- Why your loss is not going down — the flat-line case this lesson did not cover.
Developer — Code and libraries.
Setup
pip install torch matplotlibVerified with torch 2.5.1 (CPU), matplotlib 3.9.3, Python 3.10. Runs in about fifteen seconds with no GPU and no download — the data is generated in memory. Seeds are fixed, so your numbers should match to three decimals. Small differences across BLAS or PyTorch builds are normal and do not change any shape below.
Four curves from one script
The same tiny network, the same synthetic task, four settings. Nothing here is a special case invented to make a point — these are the settings that produce each shape.
import torch, torch.nn as nn
def make_data(n, seed): # one noisy non-linear pattern
g = torch.Generator().manual_seed(seed)
X = torch.randn(n, 8, generator=g)
y = (X[:, 0] * X[:, 1] + 0.4 * X[:, 2] + 0.3 * torch.randn(n, generator=g) > 0).float()
return X, y[:, None]
def run(lr, n_train, hidden, epochs, seed=0):
Xtr, ytr = make_data(n_train, seed)
Xva, yva = make_data(600, seed + 999) # held-out data, never trained on
torch.manual_seed(seed)
net = nn.Sequential(nn.Linear(8, hidden), nn.ReLU(), nn.Linear(hidden, 1))
opt, loss_fn = torch.optim.SGD(net.parameters(), lr=lr), nn.BCEWithLogitsLoss()
train_hist, val_hist = [], []
for _ in range(epochs):
net.train()
opt.zero_grad(); loss_fn(net(Xtr), ytr).backward(); opt.step()
net.eval()
with torch.no_grad(): # both sets scored the same way, after the step
train_hist.append(loss_fn(net(Xtr), ytr).item())
val_hist.append(loss_fn(net(Xva), yva).item())
return train_hist, val_hist
runs = {"healthy": run(lr=1.0, n_train=800, hidden=32, epochs=300),
"lr far too high": run(lr=60.0, n_train=800, hidden=32, epochs=300),
"lr a bit high": run(lr=8.0, n_train=800, hidden=32, epochs=300),
"overfitting": run(lr=0.5, n_train=40, hidden=256, epochs=300)}
for name, (tr, va) in runs.items():
print(name)
for e in range(0, 300, 60):
print(f" epoch {e:3d} train {tr[e]:9.3g} val {va[e]:9.3g}")healthy epoch 0 train 0.672 val 0.678 epoch 60 train 0.275 val 0.336 epoch 120 train 0.236 val 0.329 epoch 180 train 0.215 val 0.341 epoch 240 train 0.198 val 0.353 lr far too high epoch 0 train 2.36 val 2.52 epoch 60 train nan val nan epoch 120 train nan val nan epoch 180 train nan val nan epoch 240 train nan val nan lr a bit high epoch 0 train 0.659 val 0.678 epoch 60 train 0.442 val 0.643 epoch 120 train 0.369 val 0.65 epoch 180 train 0.216 val 0.471 epoch 240 train 0.367 val 0.815 overfitting epoch 0 train 0.65 val 0.715 epoch 60 train 0.106 val 1 epoch 120 train 0.0362 val 1.31 epoch 180 train 0.0192 val 1.5 epoch 240 train 0.0125 val 1.64
Read the numbers before reading the explanation. Each block is a different illness, and each one is diagnosable from the columns alone.
The walkthrough
lr far too high reaches nan and stays there. nan means "not a number" — the loss grew so large that the arithmetic gave up. Every later epoch is wasted compute, because once weights are nan they never recover. One check for nan after each epoch saves hours; NaN loss debugging covers the other causes.
lr a bit high is the dangerous one. It never crashes. It descends, backs up, descends again — 0.442, then 0.369, then 0.216, then 0.367. That last rise appears on both lines, so it is overshooting, not memorising. Anyone reading only the final number sees "loss 0.5, not great" and reaches for a bigger model, which is the wrong repair.
overfitting splits the two lines within a few epochs. Training loss falls to 0.0125 while validation climbs to 1.64. Forty training rows against 256 hidden units is more capacity than data, so the network memorises. Best validation loss here is 0.642, reached at epoch 3 — every epoch after that was actively harmful.
Even healthy turns. Its validation loss bottoms at 0.325 around epoch 95, then drifts up to 0.371 while training keeps falling. That turning point is the whole argument for early stopping and keeping the best checkpoint. A curve is healthy up to its turn, not forever.
Why full-batch steps? Each epoch here is one gradient step on the whole dataset, which makes the curves clean and the lesson readable. Real training uses mini-batches, and that changes the noise level — see what changing batch size actually changes.
Drawing it properly
Numbers diagnose; pictures make the diagnosis obvious in one second. Append this to the same file.
import matplotlib
matplotlib.use("Agg") # no display needed on a server
import matplotlib.pyplot as plt
fig, axes = plt.subplots(1, 4, figsize=(16, 3.5), sharex=True)
for ax, (name, (tr, va)) in zip(axes, runs.items()):
ax.plot(tr, label="train")
ax.plot(va, label="validation")
ax.set_yscale("log") # a loss that explodes needs a log axis
ax.set_title(name); ax.set_xlabel("epoch")
axes[0].set_ylabel("loss"); axes[0].legend()
fig.tight_layout(); fig.savefig("curves.png", dpi=110)
print("wrote curves.png")wrote curves.png
Open curves.png. The exploding panel needs the log axis to stay on the page at all. That axis quietly changes how the other three panels look — a trap covered in how smoothing and log scales mislead you.
Common mistakes
Plotting only the training loss. The most common error in the whole of ML. Training loss alone cannot separate a model that is learning from a model that is memorising — both look like a pleasant falling line. The validation line is where the truth is.
Comparing two curves measured differently. Training loss is often averaged across an epoch while the weights are changing; validation loss is measured once at the end with fixed weights. That alone makes training loss look worse early on. This script scores both after the step, under net.eval(), so the comparison is honest. Validation loop and metric accumulation has the full set of traps here.
Throwing away the history. Keep train_hist and val_hist in a file, not in a notebook variable that dies with the kernel. Six weeks later the question becomes "was this run always like that?", and only a saved curve answers it. Experiment tracking exists for exactly this.
Reading a curve with fewer than about ten points. Three points are not a trend. Log every epoch, or every fixed number of steps, before drawing conclusions about shape.
Try it yourself
Change lr=8.0 to lr=4.0, then lr=2.0, and watch the bouncing shrink into the healthy shape. Then find the smallest hidden value that stops the overfitting run from splitting its two lines. You will have discovered by hand the trade-off in too small a model, or trained too little?.
What to learn next
- When validation loss rises but accuracy improves — the first curve that will contradict this lesson.
- Finding a learning rate — how to avoid shapes 2 and 3 before you ever plot them.
- Why your loss is not going down — the flat-line case this lesson did not cover.
Researcher — Mathematics and papers.
What the curve is actually measuring
Let $\theta_t$ be the parameters after $t$ optimisation steps, $\ell$ the per-example loss, $S$ the training sample and $D$ the data distribution. The two plotted quantities are the empirical risk
$$ \hat{R}_S(\theta_t) = \frac{1}{|S|}\sum_{(x,y)\in S} \ell(f_{\theta_t}(x), y) $$
and a Monte Carlo estimate of the population risk $R(\theta_t) = \mathbb{E}{(x,y)\sim D}\,\ell(f{\theta_t}(x), y)$ taken on a held-out sample $V$. Here $f_\theta$ is the model, $|S|$ the training-set size, and $V$ the validation set. The plotted gap $\hat{R}_V(\theta_t) - \hat{R}_S(\theta_t)$ estimates the generalisation gap, and it is an estimate with variance on the order of $\sigma^2/|V|$ — a fact the eye ignores and noisy validation curves makes concrete.
Two consequences matter for reading shapes. First, $\hat{R}_V$ is unbiased for $R$ only until you select on it; every early-stopping decision made on $V$ biases $\hat{R}_V$ downward, which is why a separate test set exists (test-set overfitting). Second, both curves are functions of $t$, not of data size — the sample-size analogue is the learning curve, a different instrument that readers routinely confuse with this one.
Divergence and the stability threshold
The exploding shape has a clean theoretical account. For gradient descent on a locally quadratic objective with Hessian $H$ and largest eigenvalue $\lambda_{\max}$, the iteration $\theta_{t+1} = \theta_t - \eta \nabla \hat{R}(\theta_t)$ contracts along an eigendirection with eigenvalue $\lambda$ only when $|1 - \eta\lambda| < 1$, giving the classical stability condition $\eta < 2/\lambda_{\max}$. Above it, the component along the top eigendirection grows geometrically — the observed exponential blow-up to overflow, then nan.
Deep networks complicate this productively. Cohen et al. (2021), Gradient Descent on Neural Networks Typically Occurs at the Edge of Stability (ICLR 2021), show that $\lambda_{\max}$ is not fixed: full-batch gradient descent drives sharpness upward until it hovers slightly above $2/\eta$, where training becomes non-monotone yet still decreases loss over long horizons. The bouncing panel of this lesson is that regime in miniature — a curve oscillating locally while trending down is not necessarily misconfigured, which is why practitioners judge trend over a window rather than over the last two points.
The overfitting shape, and where it stops being the whole story
The classical U-shaped validation curve in $t$ has the same origin as the U in model complexity: approximation error falls while an estimation/variance term rises. For gradient descent on least squares, stopping at step $t$ with rate $\eta$ shrinks the coefficient along eigendirection $\lambda_i$ by a factor $1 - (1-\eta\lambda_i)^t$, a soft-thresholding profile closely related to ridge regression with penalty roughly $1/(\eta t)$. Early stopping is not a hack; it is a regularisation path traversed in time.
The modern caveat is that the U is not universal. Over-parameterised networks can show validation loss rising and then falling again over long training — see plateaus, sudden drops and double descent and Nakkiran et al., Deep Double Descent (ICLR 2020; JSTAT 2021), whose epoch-wise variant is precisely a loss curve that violates the classical reading. Treat "validation is rising, stop now" as a strong default with a documented exception class, not as a law.
Loss versus the metric you were paid for
The plotted loss is a surrogate. It is chosen for differentiability and calibration, not because anyone cares about cross-entropy in nats. The decision metric — accuracy, recall at fixed precision, revenue — is a different function of the same $\theta_t$, and the two can move in opposite directions for entirely principled reasons. That divergence is common enough to deserve its own lesson: when validation loss rises but accuracy improves. The practical discipline is to plot both, always, and to state in advance which one the stopping rule reads.
What to learn next
- When validation loss rises but accuracy improves — the first curve that will contradict this lesson.
- Finding a learning rate — how to avoid shapes 2 and 3 before you ever plot them.
- Why your loss is not going down — the flat-line case this lesson did not cover.