Model compression

Model compression makes a trained model smaller using four levers — a smaller design, fewer bits per weight, removing weights, and training a small model to copy a big one.

By Pranay Mahendrakar
12 min read
3 reading levels
Updated 20 Aug 2026

Read these first

On this page 8

The short answer
The analogy you have already lived
Why it exists
The four levers
How they fit together
The honest part
Remember this
What to learn next

One lesson, three depths. Pick the one that fits you today — you can switch any time.

Beginner — No maths. Plain English.

The short answer

Model compression means making a trained model smaller without making it much worse.

The analogy you have already lived

Think about packing a suitcase for a flight with a weight limit.

You have four moves. Take fewer clothes. Take thinner clothes. Roll them tightly so they take less space. Or buy a small travel kit that does the job of the big bottles.

Every one of those four costs you something. Fewer clothes means less choice. A travel kit runs out sooner. You accept the cost, because the suitcase has to close.

A model going onto a phone has to close too.

Why it exists

A model that trains well is usually far bigger than it needs to be for answering questions.

During training, a network needs room to explore. It keeps many weights that end up contributing almost nothing. It stores each number to a precision far beyond what the final answer requires.

That waste is harmless in a data centre. On a phone it is the difference between an app people install and one they delete.

The four levers

One: design it smaller. Use fewer layers, or fewer units in each layer. This is decided before training, and it is by far the cheapest lever.

Two: use fewer bits per number. Store each weight in one byte instead of four. This is quantisation.

Three: throw weights away. Set the least useful weights to zero and keep the rest. This is pruning.

Four: train a small model to imitate a big one. The big model teaches the small model, and the small one ends up better than it would have been alone. This is knowledge distillation.

Each has its own lesson after this one. This lesson is the map.

How they fit together

  big trained model
        |
        |--- fewer bits per weight  ->  4x smaller, tiny quality loss
        |--- zero out weak weights   ->  smaller when zipped, needs retraining
        |--- small model copies it   ->  much smaller, needs the big one first
        |
  small model that fits on a phone

They stack. You can quantise a pruned model. You can distil into an already-small design and then quantise that. The savings multiply.

The honest part

Every lever costs quality. Sometimes the cost is too small to measure, and sometimes the model falls over completely.

There is no way to know which without testing on your own task. A compression setting that a blog post calls "lossless" may break your model badly. Measure, on your data, every time. This is not a step you can skip by trusting a table.

The second honest thing: smaller does not always mean faster. A compressed model can run more slowly than the original. It happens when the hardware has no fast path for the new format. You will see that happen in the next lesson.

Remember this

Four levers: smaller design, fewer bits, fewer weights, copy a bigger model.
They stack, and the savings multiply.
Every one costs some quality, so measure it on your own task.

What to learn next

Quantisation in practice — the lever you will reach for first.
Pruning — why zeroed weights do not shrink a file on their own.
Knowledge distillation — training a small model to copy a big one.

Developer — Code and libraries.

Setup

bash

pip install torch scikit-learn

The CPU build of PyTorch is enough. No GPU, no downloads beyond the libraries — the dataset ships inside scikit-learn.

All four levers on one model

This trains a small network on the bundled digits dataset, then applies each lever and prints size and accuracy for each. It takes under ten seconds on a laptop.

compress.py

import copy, gzip, os, torch, torch.nn as nn, torch.nn.functional as F
import torch.nn.utils.prune as prune
from sklearn.datasets import load_digits
from sklearn.model_selection import train_test_split

torch.set_num_threads(1)          # one thread, so timings and results are comparable
X, y = load_digits(return_X_y=True)
Xtr, Xte, ytr, yte = train_test_split(X / 16.0, y, test_size=0.3, random_state=0, stratify=y)
Xtr = torch.tensor(Xtr, dtype=torch.float32); ytr = torch.tensor(ytr, dtype=torch.long)
Xte = torch.tensor(Xte, dtype=torch.float32); yte = torch.tensor(yte, dtype=torch.long)

def mlp(h):
    return nn.Sequential(nn.Linear(64, h), nn.ReLU(), nn.Linear(h, h), nn.ReLU(), nn.Linear(h, 10))

def fit(model, epochs=30, lr=1e-3):
    opt = torch.optim.Adam(model.parameters(), lr=lr)
    for _ in range(epochs):
        for i in torch.randperm(len(Xtr)).split(64):
            opt.zero_grad(); F.cross_entropy(model(Xtr[i]), ytr[i]).backward(); opt.step()
    return model.eval()

def acc(m):
    with torch.no_grad():
        return (m(Xte).argmax(1) == yte).float().mean().item()

def report(label, model, path):
    torch.save(model.state_dict(), path)
    raw = os.path.getsize(path)
    with open(path, "rb") as f, gzip.open(path + ".gz", "wb", 9) as g:
        g.write(f.read())                      # what the download would actually cost
    print("%-34s %8.1f %11.1f %10.4f" % (label, raw / 1024, os.path.getsize(path + ".gz") / 1024, acc(model)))

print("%-34s %8s %11s %10s" % ("model", "KB", "KB zipped", "accuracy"))

torch.manual_seed(0); base = fit(mlp(256))
report("baseline, 256 wide", base, "a.pt")

torch.manual_seed(0); small = fit(mlp(32))                       # lever 1: a smaller design
report("smaller design, 32 wide", small, "b.pt")

q = torch.ao.quantization.quantize_dynamic(base, {nn.Linear}, dtype=torch.qint8)   # lever 2
report("baseline, int8 weights", q, "c.pt")

pruned = copy.deepcopy(base)                                     # lever 3
lins = [l for l in pruned if isinstance(l, nn.Linear)]
for l in lins:
    prune.l1_unstructured(l, name="weight", amount=0.8)          # zero the 80% smallest weights
torch.manual_seed(1); fit(pruned, epochs=15, lr=5e-4)            # let the survivors take up the slack
for l in lins:
    prune.remove(l, "weight")
report("baseline, 80% of weights zeroed", pruned, "d.pt")

qp = torch.ao.quantization.quantize_dynamic(pruned, {nn.Linear}, dtype=torch.qint8)
report("pruned 80% AND int8", qp, "e.pt")

Output

model                                    KB   KB zipped   accuracy
baseline, 256 wide                    334.1       308.6     0.9648
smaller design, 32 wide                15.6        13.6     0.9556
baseline, int8 weights                 88.3        73.5     0.9648
baseline, 80% of weights zeroed       334.1        95.3     0.9778
pruned 80% AND int8                    88.3        30.5     0.9778

Five rows, five lessons

The smaller design is the biggest single win. 334 KB down to 15.6 KB — 21 times smaller — for 0.9 points of accuracy. No clever technique in this whole section beats "did you need 256 units?". Ask that question before you reach for anything else.

int8 cost nothing here. Identical accuracy, 3.8 times smaller. That is the usual result at 8 bits, and the next lesson shows where it stops being free.

Pruning did not shrink the file at all. Look at the third column: 334.1 KB, unchanged. A weight set to zero is still a 4-byte zero on disk. Pruning pays off only when something downstream exploits the zeros — a zip file, a sparse storage format, or sparse-aware hardware. Zipped, the pruned model is 95.3 KB against the baseline's 308.6 KB.

Pruning made the model better. 0.9778 against 0.9648. This is real and it is not magic. Removing 80% of the weights and retraining acts like a strong regulariser on a small dataset. Do not expect it on a large one.

The levers stack. Pruned and quantised together: 30.5 KB zipped, ten times smaller than the baseline download, with higher accuracy. That combination is the normal shipping configuration.

Which lever to reach for

Lever	Typical size win	Quality cost	When to use it
Smaller design	5–50x	Real, and you control it	Always ask first
Quantisation, 8-bit	4x	Usually near zero	Almost always
Quantisation, 4-bit	8x	Small but real	Large models with redundancy
Pruning	2–10x zipped	Zero after retraining, up to a point	When you can retrain
Distillation	10–100x	Depends on the teacher	When you already have a big model

Common mistakes

Compressing before you know the target. "As small as possible" is not a requirement. Find the device's real memory and download limits first, then compress to fit them and stop.

Reporting the uncompressed size. Users download the zipped file. Report both, as the script above does, or you will over-report the pruning benefit and under-report everything else.

Skipping the retraining step after pruning. Delete the fit(pruned, ...) line and rerun. Accuracy drops from 0.9778 to about 0.86. The retrain is not optional.

Assuming smaller means faster. File size and latency are different measurements with different causes. Measure both, and see latency and throughput.

Testing on the training set. Compression damage shows up on held-out data first. Every accuracy number above is on the test split.

Try it yourself

Change the pruning amount from 0.8 to 0.95 and rerun. Then remove the retraining line and rerun again. You now have four numbers describing one decision, which is what an honest compression report looks like.

What to learn next

Quantisation in practice — the lever you will reach for first.
Pruning — why zeroed weights do not shrink a file on their own.
Knowledge distillation — training a small model to copy a big one.

Researcher — Mathematics and papers.

The four families, formally

Let $f_\theta$ be a trained network with parameters $\theta \in \mathbb{R}^n$. Compression seeks $\hat{f}_{\hat\theta}$ minimising a resource cost $R$ subject to a quality constraint:

$$ \min_{\hat\theta} \; R(\hat{f}{\hat\theta}) \quad \text{s.t.} \quad \mathbb{E}{x \sim \mathcal{D}}!\left[\ell!\left(\hat{f}{\hat\theta}(x), f\theta(x)\right)\right] \le \epsilon $$

$R$ — the resource being minimised: bytes, multiply-accumulates, or joules. These three are not interchangeable, and optimising one can worsen another.
$\mathcal{D}$ — the deployment distribution, which is not the training distribution.
$\ell$ — a divergence between the compressed and original outputs. Note the constraint is written against the teacher's outputs, not against labels; that choice is what makes distillation a member of this family.

The four families differ in what they are allowed to change:

Family	Changes	Retraining required
Architecture search	the function class	Full training
Quantisation	the numeric representation of $\theta$	None to light
Pruning	the support of $\theta$	Light to full
Distillation	both class and parameters	Full training of the student

Why compression works at all

The empirical fact underlying everything here: trained networks are heavily over-parameterised relative to the function they implement.

Denil et al. (2013), Predicting Parameters in Deep Learning, showed that 95% of the weights in some networks can be predicted from the remaining 5%, implying the weight matrix is close to low-rank. Frankle and Carbin (2019), The Lottery Ticket Hypothesis, showed that dense networks contain sparse subnetworks which, trained from the same initialisation, match the dense network's accuracy.

The mechanistic reading is that over-parameterisation is a training aid rather than a representational requirement. It smooths the loss landscape and makes gradient descent find good solutions. Once found, the solution does not need the scaffolding. This is why compression after training works far better than training the small model directly — which is exactly what the distillation numbers show.

The pipeline that became standard

Han, Mao and Dally (2016), Deep Compression, established the ordering that most toolchains still follow:

Prune to sparsity $s$, then retrain the survivors.
Quantise the survivors, with cluster-shared weights.
Entropy-code the result, typically Huffman.

They reported 35–49x reduction on AlexNet and VGG-16 with no top-1 accuracy loss. The paper's important structural claim is that these stages compose without interfering — pruning does not make quantisation harder, and both leave the residual redundancy that entropy coding removes.

The ordering matters. Pruning before quantisation is standard, because pruning perturbs the weight distribution and quantisation ranges should be calibrated after that perturbation, not before.

What the size metric hides

Three different numbers are all called "model size", and papers move between them without saying so:

Parameter count $n$. Hardware-independent, and a poor proxy for anything.
Serialised bytes. What the user downloads. Depends on dtype, container format and entropy coding.
Peak resident memory. Weights plus activations plus the runtime's own arena. For a convolutional network at batch 1, activations frequently exceed weights, so a 4x weight reduction can produce a much smaller improvement in peak RSS.

Latency is a fourth axis, correlated with none of them reliably. Unstructured sparsity reduces parameter count and serialised bytes while leaving dense-kernel latency completely unchanged, which is the single most common misreading in the compression literature.

Current practice for large models

For transformer-scale models the mix has shifted:

Quantisation dominates. GPTQ (Frantar et al., 2022) and AWQ (Lin et al., 2023) do weight-only post-training quantisation to 4 bits with second-order or activation-aware error compensation. See quantisation in practice.
Structured pruning has largely displaced unstructured, because whole-head and whole-channel removal maps onto dense kernels. LLM-Pruner (Ma et al., 2023) and SliceGPT (Ashkboos et al., 2024) are representative.
Distillation is used at pretraining scale. DistilBERT (Sanh et al., 2019) retained about 97% of BERT's GLUE score at 40% fewer parameters and 60% faster inference.
Low-rank adaptation (LoRA) is not compression of the base model, but it is why one base model plus many small adapters is now cheaper to ship than many fine-tuned models.

Reading

Han, Mao and Dally, Deep Compression, ICLR 2016 — arxiv.org/abs/1510.00149
Denil et al., Predicting Parameters in Deep Learning, NeurIPS 2013 — arxiv.org/abs/1306.0543
Frankle and Carbin, The Lottery Ticket Hypothesis, ICLR 2019 — arxiv.org/abs/1803.03635
Cheng et al., A Survey of Model Compression and Acceleration for Deep Neural Networks, 2017 — arxiv.org/abs/1710.09282
Blalock et al., What is the State of Neural Network Pruning?, MLSys 2020 — a sobering audit of how rarely pruning papers are comparable.

What to learn next

Quantisation in practice — the lever you will reach for first.
Pruning — why zeroed weights do not shrink a file on their own.
Knowledge distillation — training a small model to copy a big one.

🟢 Beginner — No maths. Plain English.

The short answer#

The analogy you have already lived#

Why it exists#

The four levers#

How they fit together#

The honest part#

Remember this#

What to learn next#

🟡 Developer — Code and libraries.

Setup#

All four levers on one model#

Five rows, five lessons#

Which lever to reach for#

Common mistakes#

Try it yourself#

What to learn next#

🔴 Researcher — Mathematics and papers.

The four families, formally#

Why compression works at all#

The pipeline that became standard#

What the size metric hides#

Current practice for large models#

Reading#

What to learn next#

Beginner — No maths. Plain English.

The short answer

The analogy you have already lived

Why it exists

The four levers

How they fit together

The honest part

Remember this

What to learn next

Developer — Code and libraries.

Setup

All four levers on one model

Five rows, five lessons

Which lever to reach for

Common mistakes

Try it yourself

What to learn next

Researcher — Mathematics and papers.

The four families, formally

Why compression works at all

The pipeline that became standard

What the size metric hides

Current practice for large models

Reading

What to learn next