Writing a validation loop that reports the truth

Setup#

pip install torch

Outputs captured with torch 2.5.1 on CPU; deterministic under the seed shown.

A correct loop, with the trap measured#

100 samples in batches of 32 make batches of 32, 32, 32 and 4 — deliberately uneven.

import torch
import torch.nn as nn
from torch.utils.data import DataLoader, TensorDataset

torch.manual_seed(0)
X = torch.randn(100, 8)
y = torch.randint(0, 2, (100,))
loader = DataLoader(TensorDataset(X, y), batch_size=32)   # batch sizes: 32, 32, 32, 4

model = nn.Linear(8, 2)
loss_fn = nn.CrossEntropyLoss()

model.eval()                                   # dropout off, batchnorm frozen
naive_sum, loss_sum, correct, n = 0.0, 0.0, 0, 0
with torch.no_grad():                          # no graph: honest speed, no memory growth
    for xb, yb in loader:
        out = model(xb)
        batch_loss = loss_fn(out, yb)
        naive_sum += batch_loss.item()                 # wrong: a 4-sample batch counts as much as a 32-sample one
        loss_sum += batch_loss.item() * len(xb)        # right: weight every batch by its size
        correct += (out.argmax(dim=1) == yb).sum().item()
        n += len(xb)

print(f"mean of batch means: {naive_sum / len(loader):.4f}")
print(f"true per-sample loss: {loss_sum / n:.4f}")
print(f"accuracy: {correct}/{n} = {correct / n:.1%}")

mean of batch means: 0.7542
true per-sample loss: 0.8033
accuracy: 53/100 = 53.0%

The two loss numbers disagree by 6% — from nothing but arithmetic. Four samples got the voting power of thirty-two. With an unlucky last batch the gap grows, and it changes run to run as the data order changes, adding noise exactly where you want stability.

The three guards, one by one#

model.eval() — switches layers with two personalities into exam behaviour: dropout stops dropping, batchnorm uses its stored statistics instead of the current batch's. Skipping this makes validation noisy and wrong. Details live in train and eval mode.

torch.no_grad() — tells autograd not to build the graph for anything inside. Validation needs no gradients; without this guard it pays graph-building time and holds activations in memory for a backward pass that never comes.

Totals, then one division — the fix for the class-average trap. Accuracy above never touches batch averages at all: it counts correct predictions and samples, then divides once.

.item() — converts the loss tensor to a plain float on the spot. Accumulating the tensor itself drags its history along and leaks memory — the mechanism is dissected in memory that grows every epoch.

Common mistakes#

Forgetting model.train() afterwards. The loop above leaves the model in eval mode. Resume training without switching back and dropout stays dead while batchnorm stops updating — training continues, quietly wrong. Pair every eval() with a train() when the loop returns.

Metrics that cannot be averaged at all. Loss and accuracy accumulate as totals. Precision, recall, F1 and AUC do not: the F1 of the whole set is not any weighted average of per-batch F1s. Accumulate raw counts (or all predictions) and compute once — or use TorchMetrics, which exists for exactly this.

Validating with augmented data. Random crops and flips belong to training. A validation set that changes every epoch measures a moving target; keep its transforms deterministic.

Peeking every few batches with a partial validation. Validating on the first two batches "for speed" reports a different population each time the loader order changes. Validate on the full set, less often, instead.

Try it yourself#

Set batch_size=30 (batches of 30, 30, 30, 10) and rerun. Predict first whether the naive number moves closer to the truth or further away, then check. Then add drop_last=True to the loader and explain what the naive method now silently ignores.

What to learn next#

• TorchMetrics — the accumulate-then-compute pattern, packaged and battle-tested.
• Train and eval mode — what model.eval() actually switches.
• Model evaluation — which metric to report in the first place.