Learn AI Create the Future — with Pranay
Main site

Baselines and Choosing a Model

Freeze, fine-tune, or train from scratch?

With a pretrained model available, the real choice is how much of it to retrain — and the answer follows from your data size and how far your task sits from the model's old one.

Read these first

On this page 5
  1. Why it exists
  2. How it works
  3. A real example you have seen
  4. Remember this
  5. What to learn next

One lesson, three depths. Pick the one that fits you today — you can switch any time.

Beginner — No maths. Plain English.

When a pretrained model exists, the decision is not whether to use it but how much of it to retrain: none, some, or all.

A restaurant hiring a cook does not teach them to hold a knife. An experienced cook arrives knowing heat, timing, and knife work — the restaurant teaches only its own menu. Training from scratch would mean hiring someone who has never cooked and starting with "this is an onion".

A pretrained model is the experienced cook: a network someone else trained on a mountain of general data. Your decision is how much retraining the new job needs.

Why it exists

Training a big model from nothing needs data and compute most teams will never have. Pretrained models turned that from a wall into a choice: start from knowledge, adapt what you must.

But "adapt" has a dial, and each setting costs differently:

  • Retrain nothing extra: fast, cheap, safe — and limited.
  • Retrain a little: the middle path, most common in practice.
  • Retrain everything: powerful, hungry, and able to destroy what it inherited.

How it works

Three settings of the dial, and the two questions that choose between them:

FREEZE      keep the whole cook, teach only the menu card
            → tiny data (hundreds of examples), similar task

FINE-TUNE   let the cook adjust their late habits to your kitchen
  (partly)  → medium data (thousands), task somewhat different

SCRATCH     hire a beginner, teach everything
            → huge data, or a task so alien the old skills mislead

question 1: how much labelled data do you have?
question 2: how far is your task from what the model knew?

Retraining everything on little data carries a danger. The model overwrites its general knowledge with your handful of examples. It is like a cook forgetting knife skills to memorise five recipes. Small data plus full retraining is the classic self-inflicted wound.

A real example you have seen

Voice assistants adapt to your accent without relearning the language. Phone cameras that recognise your face did not relearn what faces are. Both are frozen general knowledge with a thin personal layer trained on top — the dial at its first setting.

Remember this

  • The choice is a dial: freeze, fine-tune partly, or train from scratch.
  • Less data → freeze more. More distant task → retrain more.
  • Full retraining on small data destroys inherited knowledge — the classic mistake.

What to learn next

Developer — Code and libraries.

Setup

bash
pip install torch torchvision

Outputs verified with torch 2.5.1 and torchvision 0.20, CPU.

The dial, in parameter counts

The mechanics of all three strategies are requires_grad flags. This runs download-free; swapping in real pretrained weights is a one-argument change.

freeze_dial.py
import torch
from torchvision import models

# weights=None keeps this download-free. For the real thing use
# weights="IMAGENET1K_V1", a one-time ~45 MB download.
net = models.resnet18(weights=None)
net.fc = torch.nn.Linear(net.fc.in_features, 5)   # our task: 5 classes

def trainable(m):
    return sum(p.numel() for p in m.parameters() if p.requires_grad)

total = sum(p.numel() for p in net.parameters())
print(f"total parameters:            {total:,}")

# Strategy 1: freeze everything, train the new head only.
for p in net.parameters():
    p.requires_grad = False
for p in net.fc.parameters():
    p.requires_grad = True
print(f"head only:                   {trainable(net):,} trainable")

# Strategy 2: also unfreeze the last residual stage.
for p in net.layer4.parameters():
    p.requires_grad = True
print(f"last stage + head:           {trainable(net):,} trainable")

# Strategy 3: fine-tune everything.
for p in net.parameters():
    p.requires_grad = True
print(f"full fine-tune:              {trainable(net):,} trainable")

# The optimizer should only ever see what is trainable.
frozen_setup = [p for p in net.parameters() if p.requires_grad]
opt = torch.optim.AdamW(frozen_setup, lr=1e-4)
print(f"parameter tensors handed to the optimizer: {len(frozen_setup)}")
Output
total parameters:            11,179,077
head only:                   2,565 trainable
last stage + head:           8,396,293 trainable
full fine-tune:              11,179,077 trainable
parameter tensors handed to the optimizer: 62

The walkthrough

2,565 versus 11 million. Head-only training touches 0.02% of the network — which is why it works with a few hundred labelled images and finishes in minutes on CPU. The other 99.98% acts as a fixed feature extractor.

layer4 is the "late habits" of the analogy. Early convolution layers hold generic edges and textures that transfer almost anywhere; late layers hold task-specific combinations. Unfreezing from the back forward is the standard middle path, and the parameter jump — 2.5k to 8.4M — shows why it needs thousands of examples, not hundreds.

The optimizer line is the practical bug-catcher. Handing net.parameters() wholesale to the optimizer while relying on requires_grad works, but filtering explicitly documents intent and enables tricks like per-layer learning rates.

Two rates when fine-tuning fully: new head at a normal rate, inherited body at a tenth of it. The body already knows things; big steps make it forget.

Common mistakes

Full fine-tuning on 500 examples. The network happily memorises your 500 and shreds its general features doing so — catastrophic forgetting in miniature. Freeze first; unfreeze only if the frozen version demonstrably plateaus.

Forgetting the preprocessing contract. Pretrained weights assume the exact normalisation of their pretraining (for ImageNet models: specific per-channel means and stds). Feed differently scaled pixels and accuracy quietly craters with no error message.

Leaving BatchNorm in training mode with tiny batches. Frozen backbones with unfrozen BatchNorm statistics silently adapt to your little dataset and destabilise. Call .eval() on frozen sections or freeze the norm layers explicitly — details in BatchNorm in PyTorch.

Comparing strategies at one data size. The ranking flips with data volume — the entire point. If you can, plot all three against 10%, 50%, 100% of your data; the crossing points tell you what to do when more data arrives.

Try it yourself

Print the trainable-parameter count when you unfreeze layer3 as well. Then, using the ~45 MB figure for resnet18's weights, estimate download plus disk cost for resnet50 (~98 MB) — and answer: for a 400-image dataset of five product types, which dial setting do you pick, and why?

What to learn next

Researcher — Mathematics and papers.

What transfers, and from where

Yosinski et al. (2014), How transferable are features in deep neural networks? (NeurIPS), quantified layer-wise transferability with splice experiments: early layers are general, late layers specific, and transfer benefit degrades with task distance. Two additional effects they isolated: fragile co-adaptation when splitting mid-network, and a persistent generalisation boost from transferred initialisations even after long fine-tuning.

Follow-ups refined the picture:

  • Kornblith et al. (2019), Do Better ImageNet Models Transfer Better? (CVPR) — largely yes, with correlation to ImageNet accuracy, but with diminishing and task-dependent returns.
  • Raghu et al. (2019), Transfusion (NeurIPS) — for medical imaging, transfer's benefit is often re-use of low-level statistics and better initialisation scale, not high-level features; sometimes a smaller scratch model matches.
  • He, Girshick, Dollár (2019), Rethinking ImageNet Pre-training (ICCV) — with enough target data and training time, scratch matches pretraining on COCO; pretraining mainly buys speed of convergence, not final accuracy, when data is plentiful. This is the "SCRATCH" arm's empirical backing.

Formal view: hypothesis transfer and forgetting

Fine-tuning is regularised estimation around the source solution $\theta_0$. Explicitly: $\min_\theta \mathcal{L}_{\text{target}}(\theta) + \lambda \lVert \theta - \theta_0 \rVert^2$ recovers the intuition that a small learning rate and early stopping keep $\theta$ near $\theta_0$ — an implicit prior whose strength should scale inversely with target-data size. Catastrophic forgetting (McCloskey & Cohen, 1989; measured for modern nets in Kirkpatrick et al., 2017, EWC, PNAS) is the failure of this prior under aggressive updates: EWC's Fisher-weighted penalty $\sum_i F_i (\theta_i - \theta_{0,i})^2$ formalises "protect the weights that mattered".

Parameter-efficient fine-tuning

The freeze/fine-tune dichotomy has a modern third axis: adapt through low-dimensional additions while freezing the base — adapters (Houlsby et al., 2019, ICML), and LoRA (Hu et al., 2021): $\Delta W = BA$ with rank $r \ll \min(d_{in}, d_{out})$, training ~0.1–1% of parameters to near-full fine-tuning quality in LLMs. The same data-size logic applies, shifted: PEFT methods extend the "small data" regime upward because their capacity is bounded by construction. Linear probing then fine-tuning (Kumar et al., 2022, Fine-Tuning Can Distort Pretrained Features, ICLR) sequences the dial: fit the head first so full fine-tuning starts from a non-destructive gradient field — theoretical support for "freeze first, unfreeze if it plateaus".

Decision summary from the literature

Data size and domain gap remain the two load-bearing variables. Small n, small gap → freeze (linear probe). Small n, large gap → probe-then-partial-tune, or reconsider the source model. Large n, small gap → full fine-tune at low LR. Large n, large gap → fine-tune fully; scratch becomes competitive as n grows (He et al., 2019).

What to learn next