Where ImageNet pretraining stops helping
ImageNet features answer the question "what object is this", so they help enormously on tasks shaped like that and give you almost nothing on counting, measuring, or images that are not photographs.
- 10 min read
- 3 reading levels
- Updated
Read these first
On this page 5
One lesson, three depths. Pick the one that fits you today — you can switch any time.
Beginner — No maths. Plain English.
ImageNet pretraining teaches a model one question: what object is this. It helps you only as much as your question resembles that one.
You can spot your friend on a crowded railway platform in about a second. Now count how many people are on that platform. Same eyes, same crowd, and suddenly the job is slow and you lose your place twice.
Recognising and counting are different skills. Your eyes are excellent at one and poor at the other. Practice at spotting friends never makes you a fast counter.
A pretrained vision model has the same shape of talent, for the same reason. It was rewarded, a million times, for naming the main object. It was never once rewarded for counting, measuring, or judging distance.
Why this matters more than it sounds
The advice "always start from a pretrained model" is repeated so often that people stop checking whether it applies. Often it does. Sometimes it wastes a month.
ImageNet is a dataset of over a million internet photographs. Each carries one of a thousand object labels: dog breeds, kitchen tools, vehicles, birds. A model trained on it learns exactly what that reward encourages.
ImageNet taught the model: "WHAT object is in this picture?"
Works well -> Is this weld cracked or clean?
Is this leaf healthy or diseased?
Which of my 40 colleagues is this?
Works badly -> How many cells are in this image?
How far away is that wall?
Which of these two greys is darker?The left column is naming. The right column is measuring. Naming transfers; measuring often does not.
The second gap: images that are not photographs
ImageNet photographs share a lot: daylight, three colour channels, a subject roughly in the middle, ordinary lenses.
Plenty of important image data breaks all of that. Ultrasound scans. Satellite bands your eye cannot see. Microscope slides. Sound turned into a picture. Infrared from a thermal camera.
The further your images sit from a holiday photo, the less the borrowed features are worth. They are rarely worthless. They are often worth far less than the confident tutorial suggested.
The honest test, which takes an afternoon
Before you plan a fine-tuning project, run this comparison:
- Frozen pretrained features, with a small classifier on top.
- Raw pixels, with the same small classifier on top.
- One or two numbers you compute by hand from the image, with the same classifier.
Option 1 must beat options 2 and 3 by a comfortable margin. If it does not, the pretrained model is not helping you. The developer section runs exactly this test, and the hand-made number wins.
Remember this
- ImageNet features encode what object is present, because that is what they were rewarded for.
- Tasks about counting, measuring or exact values get much less benefit.
- The further your images are from ordinary photographs, the smaller the benefit. Measure it before you build on it.
What to learn next
- Domain adaptation for vision — closing a gap between the images you trained on and the images you get.
- Few-shot image classification — the regime where pretraining is genuinely irreplaceable.
- Vision transformers — the architecture behind most domain-specific backbones today.
Developer — Code and libraries.
Setup
pip install torch torchvision scikit-learnWritten and run against torch 2.13.0 (CPU), torchvision 0.28.0, scikit-learn 1.7.2. Downloads the 44.7 MB ResNet-18 checkpoint. Total run time on a laptop CPU: about 17 seconds.
A task ImageNet was never rewarded for
The images below are black canvases with between one and five white discs. Every disc is the same size. The task is to say how many there are.
import numpy as np, torch, torch.nn as nn, torch.nn.functional as F
from sklearn.linear_model import LogisticRegression
from torchvision.models import resnet18, ResNet18_Weights
rng = np.random.default_rng(0); torch.manual_seed(0)
def canvas(k):
img = np.zeros((64, 64), "float32")
yy, xx = np.ogrid[:64, :64]
placed = []
while len(placed) < k: # keep discs apart so counting stays well defined
r, c = rng.integers(8, 56, 2)
if all((r-a)**2 + (c-b)**2 > 200 for a, b in placed):
placed.append((r, c))
img[(yy-r)**2 + (xx-c)**2 < 16] = 1.0
return img
def make(n):
lab = rng.integers(1, 6, n) # 1 to 5 discs
return torch.tensor(np.stack([canvas(k) for k in lab])).unsqueeze(1), torch.tensor(lab - 1)
Xtr, ytr = make(500); Xte, yte = make(300)
bb = resnet18(weights=ResNet18_Weights.DEFAULT); bb.fc = nn.Identity(); bb.eval()
mean = torch.tensor([0.485,0.456,0.406]).view(1,3,1,1)
std = torch.tensor([0.229,0.224,0.225]).view(1,3,1,1)
def feats(t):
z = (F.interpolate(t.repeat(1,3,1,1), size=112, mode="bilinear", align_corners=False) - mean) / std
with torch.no_grad():
return torch.cat([bb(z[i:i+50]) for i in range(0, len(z), 50)])
Ftr, Fte = feats(Xtr), feats(Xte)
print("ImageNet-feature probe :", round(LogisticRegression(max_iter=3000).fit(Ftr, ytr).score(Fte, yte), 3))
Ptr, Pte = Xtr.flatten(1), Xte.flatten(1) # every pixel as a feature
print("raw-pixel probe :", round(LogisticRegression(max_iter=3000).fit(Ptr, ytr).score(Pte, yte), 3))
Str = Xtr.sum((1, 2, 3)).unsqueeze(1) # one hand-made number: how much white ink
Ste = Xte.sum((1, 2, 3)).unsqueeze(1)
print("one number (ink total) :", round(LogisticRegression(max_iter=3000).fit(Str, ytr).score(Ste, yte), 3))ImageNet-feature probe : 0.903 raw-pixel probe : 0.57 one number (ink total) : 1.0
Identical on repeated runs with the seed fixed.
Read those three lines slowly
An 11.7-million-parameter network, trained for days on a million photographs, scored 0.903. One number that a beginner could compute with image.sum() scored 1.0.
This is not a trick against ResNet. The features are doing real work. 0.903 sits far above the raw-pixel baseline of 0.57, so they are not noise. They are a good representation of the wrong thing. ImageNet rewards "there is a disc-like object present". Nothing in that reward asks "how many".
The raw-pixel probe at 0.57 shows why the comparison matters. A linear model over 4,096 pixels cannot count either. Disc positions vary, and a linear function of pixel values cannot group them. Two of the three baselines fail. Only the one built from knowledge of the task succeeds.
The lesson generalises, and the size of the gap does not. On your data the pretrained probe may win comfortably. Run the three baselines and find out, rather than assuming.
When does ImageNet pretraining pay off? A checklist
| Your situation | Expect from ImageNet pretraining |
|---|---|
| Photographs, naming an object or condition | Large benefit, especially under a few thousand images |
| Photographs, fine-grained species or parts | Moderate benefit; the classes were not in ImageNet |
| Non-photographic images (ultrasound, radar, spectrograms) | Small benefit, mostly from the earliest layers |
| Counting, measuring, geometry, exact colour | Little benefit; often beaten by a hand-made feature |
| Hundreds of thousands of labelled target images | Faster convergence, similar final accuracy |
That last row is the finding of He, Girshick and Dollár, discussed in the researcher section. It surprises people who assume pretraining always raises the ceiling.
Common mistakes
Reporting the pretrained number with no baseline. "Our fine-tuned ResNet gets 91%" means nothing on its own. Fix: publish the raw-pixel and hand-made-feature baselines beside it. Occasionally you will delete the deep model and ship the baseline.
Assuming grey-to-RGB conversion is free. Repeating one channel three times, as feats does above, feeds the model something no ImageNet photo looked like. Every colour-opponent filter in conv1 sees zero signal. It works, and it costs you.
Resizing a 2048 x 2048 scan down to 224. The lesion you care about may be twenty pixels wide. After resizing it is two. Fix: tile the image and classify tiles, rather than shrinking the finding away.
Reaching for a bigger backbone when the features are wrong. ResNet-50 instead of ResNet-18 does not add counting ability. Fix: change the representation or the task framing, not the parameter count.
Try it yourself
Add a fourth baseline: count connected white regions with scipy.ndimage.label and feed that single integer to the same classifier. Then ask which of your four models you would rather debug at 2am.
What to learn next
- Domain adaptation for vision — closing a gap between the images you trained on and the images you get.
- Few-shot image classification — the regime where pretraining is genuinely irreplaceable.
- Vision transformers — the architecture behind most domain-specific backbones today.
Researcher — Mathematics and papers.
Pretraining helps convergence more reliably than it helps accuracy
He, Girshick and Dollár (2019), Rethinking ImageNet Pre-training (ICCV), trained COCO detection and segmentation models from random initialisation. They matched their ImageNet-pretrained counterparts on the pretrained recipe's own hyperparameters. The one change was more iterations, so the random models could converge. The result held with deeper and wider models, across tasks and metrics, and with only 10% of COCO.
Their conclusion is precise and worth restating. ImageNet pretraining speeds up early convergence. It does not necessarily provide regularisation or improve final target-task accuracy. They also note the regime where it still matters. Target data can be small enough that training from scratch is impossible.
The medical-imaging case
Raghu et al. (2019), Transfusion: Understanding Transfer Learning for Medical Imaging (NeurIPS), evaluated two large medical tasks. Transfer offered little benefit. Much smaller purpose-built models performed comparably to ImageNet architectures. Their weight-transfusion experiments localise the effect. Reuse is concentrated in the lowest layers. Part of the apparent benefit comes from over-parameterisation rather than sophisticated feature reuse.
The constructive reading: the lowest layers transfer even across a large domain gap. Gabor-like edge and blob filters are close to universal. Everything above them is source-specific. This is the empirical basis for the "tune early layers under corruption shift" result of Lee et al. (2023).
Domain-specific pretraining as the answer
Where the domain gap is structural, the field's response has been to pretrain inside the domain:
| Backbone | Pretraining data | Method |
|---|---|---|
| RadImageNet (Mei et al., 2022, Radiology: AI) | 1.35M CT/MRI/ultrasound images | Supervised |
| RETFound (Zhou et al., 2023, Nature) | 1.6M unlabelled fundus images | Masked autoencoding |
| SatMAE (Cong et al., 2022) | Multispectral / temporal satellite imagery | Masked autoencoding |
RETFound is the cleanest illustration of the modern pattern. Self-supervised pretraining removes the need for domain labels. Those labels are what made ImageNet-style supervised pretraining impossible in these fields. Zhou et al. report the adapted model outperforming comparison models on sight-threatening eye disease with fewer labelled examples.
Is the ceiling ImageNet, or supervision?
Two further lines complicate the simple story.
Kornblith et al. (2019) show ImageNet accuracy predicting transfer accuracy well overall. The correlation weakens on fine-grained datasets whose categories ImageNet never contained. Those classes were absent rather than rare, so no amount of source accuracy buys them.
Self-supervised pretraining (SimCLR, MoCo, MAE, DINOv2) removes the thousand-way label bottleneck. Oquab et al. (2023), DINOv2, report frozen features competitive with weakly-supervised alternatives on classification, segmentation and depth. That includes the dense and geometric tasks where ImageNet-supervised features are weakest. The limitation this lesson describes is therefore partly a limitation of the objective, not of pretraining as such.
Papers
- He, Girshick and Dollár, Rethinking ImageNet Pre-training, 2019 — arxiv.org/abs/1811.08883
- Raghu et al., Transfusion: Understanding Transfer Learning for Medical Imaging, 2019 — arxiv.org/abs/1902.07208
- Kornblith et al., Do Better ImageNet Models Transfer Better?, 2019 — arxiv.org/abs/1805.08974
- Zhou et al., A foundation model for generalizable disease detection from retinal images, Nature 2023 — nature.com/articles/s41586-023-06555-x
- Oquab et al., DINOv2: Learning Robust Visual Features without Supervision, 2023 — arxiv.org/abs/2304.07193
What to learn next
- Domain adaptation for vision — closing a gap between the images you trained on and the images you get.
- Few-shot image classification — the regime where pretraining is genuinely irreplaceable.
- Vision transformers — the architecture behind most domain-specific backbones today.