Finding label and target bugs

Setup#

pip install torch

Captured with torch 2.5.1 on CPU; seeded and deterministic.

The one label bug that IS loud#

Class labels numbered 1..N — from a spreadsheet, an R export, a label tool that counts from one — meeting a model that counts from zero:

import torch
import torch.nn as nn

logits = torch.randn(4, 3)                   # a 3-class model: valid labels are 0, 1, 2
labels = torch.tensor([1, 3, 2, 1])          # someone's labels run 1..3, not 0..2
loss = nn.CrossEntropyLoss()(logits, labels)

IndexError: Target 3 is out of bounds.

Merciful, when it fires. The cruel variant: labels 1..3 with num_classes=4 — every label legal, class 0 unused, no error ever, and the model quietly learns a shifted world. On CUDA this error also wears a scarier costume (device-side assert triggered, reported on some unrelated later line); rerun on CPU for the honest message, a trick from the device lesson.

Audit 1: legal values#

import torch

labels = torch.tensor([1, 3, 2, 1, 2, 3, 3, 1])      # suspect labels, straight from a CSV
num_classes = 3

print("min:", labels.min().item(), " max:", labels.max().item())
print("counts per value:", torch.bincount(labels).tolist())
assert labels.min() >= 0 and labels.max() < num_classes, \
    f"labels must be 0..{num_classes - 1}, found {labels.min()}..{labels.max()}"

min: 1  max: 3
counts per value: [0, 3, 2, 3]
AssertionError: labels must be 0..2, found 1..3

Three lines of printing expose the range, the per-class counts (a zero count for class 0 is the off-by-one fingerprint), and the assert turns tomorrow's silent version of this bug into today's loud one. Leave it in the pipeline permanently.

Audit 2: is the pairing intact?#

The subtle killer — features and labels each fine, the alignment between them broken (one side shuffled, sorted, or filtered without the other). The test: destroy the pairing on purpose and compare.

import torch
import torch.nn as nn

torch.manual_seed(0)
X = torch.randn(400, 12)
y_real = (X[:, 0] + X[:, 5] > 0).long()
y_broken = y_real[torch.randperm(len(y_real))]   # deliberately destroy the pairing

def quick_fit(X, y, steps=300):
    torch.manual_seed(1)                         # same start for both runs
    model = nn.Linear(12, 2)
    opt = torch.optim.Adam(model.parameters(), lr=0.05)
    for _ in range(steps):
        opt.zero_grad()
        nn.CrossEntropyLoss()(model(X), y).backward()
        opt.step()
    return (model(X).argmax(dim=1) == y).float().mean().item()

print(f"accuracy, real pairing:      {quick_fit(X, y_real):.1%}")
print(f"accuracy, shuffled pairing:  {quick_fit(X, y_broken):.1%}")

accuracy, real pairing:      99.3%
accuracy, shuffled pairing:  60.5%

A big gap means the true pairing carries real signal — healthy. The alarming outcome is the other one: if your real labels score no better than deliberately scrambled ones, your "real" pairing is as good as random, and the bug lives wherever features and labels were last joined. (Scrambled still beats 50% here because a 400-sample memoriser picks up crumbs — that gap shrinks on held-out data.)

Common mistakes#

Sorting one side. sorted(image_paths) next to labels read in CSV order — alignment destroyed at birth. Keep features and labels in one structure (one dataframe, one list of pairs) so they cannot be reordered separately.

Two label mappings. Train builds {"cat": 0, "dog": 1} from its own folder scan; validation's scan meets an extra folder and maps differently. Build the class-to-index mapping once, save it, load it everywhere.

Regression targets in mixed units. Half the rows in lakhs, half in rupees; or unnormalised targets producing huge losses that read like a learning-rate problem. Print y.min(), y.max(), y.mean() — the audit-1 habit works for regression too.

Trusting accuracy on the training set. Memorisation hides label noise: the one-batch test passes with completely random labels. Only held-out performance, and audits like these, see label problems.

Try it yourself#

Take audit 2 and evaluate both models on 200 fresh samples generated by the same rule, instead of the training X. Predict both numbers first. The shuffled model should now land near 50% — write one sentence on why the crumbs vanished.

What to learn next#

• Data labelling — where labels come from, and why they arrive imperfect.
• Overfit a single batch first — the machinery test to run alongside these data audits.
• Train-test split — the splitting discipline that keeps evaluations honest.