Quantisation in practice
Quantisation stores each weight in fewer bits, giving a 4x smaller model at 8 bits for almost no quality loss, and a cliff you will fall off below 4 bits.
- 16 min read
- 3 reading levels
- Updated
Read these first
On this page 8
One lesson, three depths. Pick the one that fits you today — you can switch any time.
Beginner — No maths. Plain English.
The short answer
Quantisation stores each of a model's numbers using fewer bits. The model gets smaller, and the answers get very slightly rougher.
The analogy you have already lived
Watch the weighing scale at a vegetable shop. It reads 1.2 kg. It does not read 1.20437 kg.
The scale rounds to the nearest hundred grams. Nobody has ever complained. The extra digits were never useful for buying tomatoes.
A model's weights carry the same kind of pointless precision. Each one is stored to about seven decimal places. The answers do not need anywhere near that.
Quantisation swaps the fine scale for a coarser one. The tomatoes weigh the same.
Why it exists
A trained model stores every weight as a float32 — a number taking four bytes of space.
Four bytes for every weight adds up fast. A model with a billion weights needs four billion bytes, which is four gigabytes. No phone will hold that, and no user will download it.
Store each weight in one byte instead, and the same model needs one gigabyte. That is the whole idea.
There is a second win, and it matters more than people expect. Most of the time a model spends answering a question goes on fetching weights from memory, not on the arithmetic. Fetch a quarter as many bytes and the fetching gets faster.
How it works
Picture a ruler. Float32 is a ruler with a mark every hair's breadth. Eight-bit is a ruler with 256 marks in total.
original weights: -0.37 ...... 0.31 (any value at all)
choose 256 marks spread across that range
|----|----|----|----|
each weight snaps to the nearest mark
^
store the mark number (one byte) plus the spacing (once, for the whole layer)Two things get stored. The mark number for each weight, and one scale for the group — the distance between marks. To use the weight again, multiply the mark number by the scale.
Every weight moves a little when it snaps to its mark. That movement is the cost.
What it actually costs
At 8 bits the cost is usually too small to see. At 4 bits it is small but real. Below 4 bits, models often stop working altogether — not gradually, but suddenly.
The quantisation glossary entry has a table for this. It shows what a 7-billion-parameter language model looks like at each level.
Two honest warnings.
Small models suffer more. A large model has spare capacity to absorb the rounding. A tiny model has none.
Smaller does not always mean faster. Some hardware has no fast path for 8-bit arithmetic. There, the model unpacks the bytes back into floats before every multiply. That unpacking can cost more than it saves. You will see this happen in the developer block, on real numbers.
Where you have already used it
- Any language model you run locally with Ollama or LM Studio is quantised. The "Q4" in a filename means 4-bit.
- Your phone's camera and voice models run in 8-bit integer arithmetic on a dedicated chip.
- The "smaller download" option in translation apps is usually the same model, quantised.
Remember this
- Quantisation rounds each weight onto a coarse grid and stores the grid position.
- 8 bits gives four times smaller, at almost no quality cost.
- Below 4 bits there is a cliff, and small models reach it sooner.
What to learn next
Developer — Code and libraries.
Setup
pip install torch scikit-learnCPU PyTorch is enough. Both scripts below run in about ten seconds and download nothing.
Part 1: real 8-bit quantisation, measured
torch.ao.quantization.quantize_dynamic replaces every nn.Linear with an int8 version. Weights are converted once, up front. Activations are quantised on the fly at each call, which is where the name "dynamic" comes from.
import os, time, torch, torch.nn as nn, torch.nn.functional as F
from sklearn.datasets import load_digits
from sklearn.model_selection import train_test_split
torch.manual_seed(0)
torch.set_num_threads(1) # one thread, so the timings mean something
X, y = load_digits(return_X_y=True)
Xtr, Xte, ytr, yte = train_test_split(X / 16.0, y, test_size=0.3, random_state=0, stratify=y)
Xtr = torch.tensor(Xtr, dtype=torch.float32); ytr = torch.tensor(ytr, dtype=torch.long)
Xte = torch.tensor(Xte, dtype=torch.float32); yte = torch.tensor(yte, dtype=torch.long)
model = nn.Sequential(nn.Linear(64, 256), nn.ReLU(), nn.Linear(256, 256), nn.ReLU(), nn.Linear(256, 10))
opt = torch.optim.Adam(model.parameters(), lr=1e-3)
for _ in range(30):
for i in torch.randperm(len(Xtr)).split(64):
opt.zero_grad(); F.cross_entropy(model(Xtr[i]), ytr[i]).backward(); opt.step()
model.eval()
qmodel = torch.ao.quantization.quantize_dynamic(model, {nn.Linear}, dtype=torch.qint8)
torch.save(model.state_dict(), "fp32.pt")
torch.save(qmodel.state_dict(), "int8.pt")
with torch.no_grad():
scores_fp32 = model(Xte)
scores_int8 = qmodel(Xte)
pred_fp32 = scores_fp32.argmax(1)
pred_int8 = scores_int8.argmax(1)
print("parameters :", sum(p.numel() for p in model.parameters()))
print("float32 file : %.1f KB" % (os.path.getsize("fp32.pt") / 1024))
print("int8 file : %.1f KB" % (os.path.getsize("int8.pt") / 1024))
print("float32 accuracy : %.4f" % (pred_fp32 == yte).float().mean().item())
print("int8 accuracy : %.4f" % (pred_int8 == yte).float().mean().item())
print("predictions that changed : %d out of %d" % (int((pred_fp32 != pred_int8).sum()), len(yte)))
print("mean absolute score change : %.4f" % (scores_fp32 - scores_int8).abs().mean().item())
def latency(m, n=300):
one = Xte[:1]
with torch.no_grad():
for _ in range(30): # warm up before timing anything
m(one)
t0 = time.perf_counter()
for _ in range(n):
m(one)
return (time.perf_counter() - t0) / n * 1000
print("float32 latency : %.3f ms" % latency(model))
print("int8 latency : %.3f ms" % latency(qmodel))parameters : 85002 float32 file : 334.2 KB int8 file : 88.6 KB float32 accuracy : 0.9648 int8 accuracy : 0.9648 predictions that changed : 0 out of 540 mean absolute score change : 0.0322 float32 latency : 0.024 ms int8 latency : 0.112 ms
Both sides of the trade, in one output
The win is real. 334.2 KB down to 88.6 KB — a factor of 3.8. Not a clean 4.0, because biases and bookkeeping stay in float32.
The accuracy cost is zero here. Identical to four decimal places, and not one of the 540 test predictions changed.
But the model did change. mean absolute score change: 0.0322 is the honest measurement. The scores moved. They did not move enough to flip any prediction on this task, and on a harder task with closer margins some would have flipped. Accuracy alone hides this; always report the drift too.
The latency got worse — nearly five times worse. This surprises everyone, so read it slowly. The two latency lines are measurements on one machine and will not reproduce exactly on yours; every other line in that output will. What you should reproduce is the direction.
This model is tiny. The int8 path pays a fixed cost per call: it measures the activation range, converts activations to int8, does the integer multiply, then converts back to float. On a 85,000-parameter model that overhead dwarfs the work it saves. Dynamic quantisation pays off on large matrices, where moving weight bytes is the bottleneck. On small ones it is a straight loss.
The rule: quantisation is a size technique first and a speed technique second. Whether it is also a speed win depends on your model size and your hardware, and you find out by measuring.
Part 2: how many bits can you take away?
The first script used a library. This one does the arithmetic by hand so you can see exactly where the loss comes from, and then sweeps the bit width until the model breaks.
import copy, torch, torch.nn as nn, torch.nn.functional as F
from sklearn.datasets import load_digits
from sklearn.model_selection import train_test_split
torch.manual_seed(0); torch.set_num_threads(1)
X, y = load_digits(return_X_y=True)
Xtr, Xte, ytr, yte = train_test_split(X / 16.0, y, test_size=0.3, random_state=0, stratify=y)
Xtr = torch.tensor(Xtr, dtype=torch.float32); ytr = torch.tensor(ytr, dtype=torch.long)
Xte = torch.tensor(Xte, dtype=torch.float32); yte = torch.tensor(yte, dtype=torch.long)
model = nn.Sequential(nn.Linear(64, 256), nn.ReLU(), nn.Linear(256, 256), nn.ReLU(), nn.Linear(256, 10))
opt = torch.optim.Adam(model.parameters(), lr=1e-3)
for _ in range(30):
for i in torch.randperm(len(Xtr)).split(64):
opt.zero_grad(); F.cross_entropy(model(Xtr[i]), ytr[i]).backward(); opt.step()
model.eval()
def fake_quant(w, bits):
"""Round every weight onto a grid of 2**bits levels, then put it back as a float."""
qmax = 2 ** (bits - 1) - 1 # 127 for 8 bits, 7 for 4 bits, 1 for 2 bits
scale = w.abs().max() / qmax # one scale for the whole tensor
return torch.clamp(torch.round(w / scale), -qmax - 1, qmax) * scale
w = model[0].weight.data
print("first layer weights: min %.4f max %.4f" % (w.min(), w.max()))
for bits in [8, 4, 2]:
q = fake_quant(w, bits)
print(" %2d bits -> %4d levels, step %.5f, largest error %.5f"
% (bits, 2 ** bits, (w.abs().max() / (2 ** (bits - 1) - 1)).item(), (q - w).abs().max().item()))
print()
def accuracy(m):
with torch.no_grad():
return (m(Xte).argmax(1) == yte).float().mean().item()
print("bits bytes per weight model KB accuracy")
n = sum(l.weight.numel() for l in model if isinstance(l, nn.Linear))
print(" 32 4.00 %7.1f %.4f" % (n * 4 / 1024, accuracy(model)))
for bits in [8, 6, 4, 3, 2]:
m = copy.deepcopy(model)
with torch.no_grad():
for layer in m:
if isinstance(layer, nn.Linear):
layer.weight.copy_(fake_quant(layer.weight, bits))
print("%4d %.2f %7.1f %.4f" % (bits, bits / 8, n * bits / 8 / 1024, accuracy(m)))first layer weights: min -0.3746 max 0.3094 8 bits -> 256 levels, step 0.00295, largest error 0.00147 4 bits -> 16 levels, step 0.05352, largest error 0.02676 2 bits -> 4 levels, step 0.37461, largest error 0.18727 bits bytes per weight model KB accuracy 32 4.00 330.0 0.9648 8 1.00 82.5 0.9648 6 0.75 61.9 0.9667 4 0.50 41.2 0.9685 3 0.38 30.9 0.9593 2 0.25 20.6 0.1667
The cliff
Read the accuracy column from the top. 0.9648, 0.9648, 0.9667, 0.9685, 0.9593, 0.1667.
Nothing warns you. Three bits is fine. Two bits is a coin toss — with ten classes, 0.1667 is barely above guessing. Quality does not decay smoothly as you remove bits. It holds, and then it collapses.
That is the single most important practical fact about quantisation. You cannot interpolate. Measuring at 8 bits and 4 bits tells you nothing reliable about 3.
The small rises at 6 and 4 bits are noise, not improvement. On 540 test images, one image is 0.185 percentage points. A change of 0.0037 is two images. Do not build a story on it.
Why the error triples every step down
Look at the step sizes. Halving the number of levels doubles the spacing, and the largest rounding error is half a step. From 8 bits to 4 bits, the largest error goes from 0.0015 to 0.0268 — about eighteen times worse for four bits saved.
The weights themselves span only -0.37 to 0.31. At 2 bits the grid step is 0.375, so every weight smaller than about 0.19 in size rounds to exactly zero. Most of the weights are smaller than that. The model stops being a model.
Common mistakes
Trusting a number from a blog post. "int8 is lossless" is true for many models and false for yours until you check. Run your own evaluation set.
Reporting only accuracy. Accuracy is a coarse instrument. Report the score drift as well, as the first script does, or you will ship a change you cannot see until users find it.
Quantising with one scale for the whole model. The script above uses one scale per tensor, which is already better than one per model. Production tools use one scale per output channel or per group of 64–128 weights, and the difference at 4 bits is large.
Assuming the speed-up. The first script measures a slowdown. Benchmark on the target device, in the target runtime, with the target thread count.
Quantising a model that is already small. A 5 MB model saving 3.5 MB rarely justifies the risk. Spend the effort on the biggest weights first.
Try it yourself
Change scale = w.abs().max() / qmax to compute one scale per output row, using w.abs().max(dim=1, keepdim=True).values. Rerun the sweep. On this model that lifts 3 bits above the float32 baseline, and drags 2 bits from broken up to roughly 0.76 — bad, but no longer random. That one change is most of what separates a toy quantiser from a real one.
What to learn next
Researcher — Mathematics and papers.
Affine quantisation
The standard mapping from a real tensor $r$ to a $b$-bit integer tensor $q$ is affine:
$$ q = \operatorname{clamp}!\left(\left\lceil \frac{r}{s} \right\rfloor + z,\; q_{\min},\; q_{\max}\right), \qquad \hat{r} = s\,(q - z) $$
- $s \in \mathbb{R}^{+}$ — the scale, the real-valued width of one integer step.
- $z \in \mathbb{Z}$ — the zero point, the integer that represents exact zero. Setting $z = 0$ gives symmetric quantisation; leaving it free gives asymmetric.
- $q_{\min}, q_{\max}$ — the representable range, $[-2^{b-1}, 2^{b-1}-1]$ for signed $b$-bit.
- $\lceil \cdot \rfloor$ — round to nearest, ties to even.
- $\hat{r}$ — the dequantised value. The error $\hat{r} - r$ is bounded by $s/2$ absent clipping.
For a range $[r_{\min}, r_{\max}]$ the parameters follow directly:
$$ s = \frac{r_{\max} - r_{\min}}{q_{\max} - q_{\min}}, \qquad z = q_{\min} - \left\lceil \frac{r_{\min}}{s} \right\rfloor $$
Symmetric per-channel quantisation of weights, paired with asymmetric per-tensor quantisation of activations, is what almost all toolchains default to. Jacob et al. (2018), Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference, give the integer-only matrix multiply that makes this executable on hardware with no floating-point unit at all.
The error model, and where it fails
Treating quantisation error as additive uniform noise $e \sim \mathcal{U}(-s/2, s/2)$ gives variance $s^2/12$, the classic result from signal processing. For a dot product of length $n$ with independent errors, output error variance grows as $n s^2 / 12$, so error scales as $s\sqrt{n}$: wider layers are more sensitive at a fixed bit width.
This model breaks in two places that matter.
Outliers. $s$ is set by $\max |r|$, so a single large weight inflates the step for every other weight in the group. Activation outliers in transformers are the extreme case: Dettmers et al. (2022), LLM.int8(), found systematic outlier features appearing beyond roughly 6.7B parameters whose magnitudes are 20x the rest, and whose naive quantisation destroys the model. Their fix is a mixed-precision decomposition keeping outlier dimensions in fp16.
Error is not independent across layers. It compounds through depth, and the compounding is what produces the cliff in the developer block rather than a smooth decline.
Granularity, which matters more than bit width below 8
| Granularity | Scales stored | Typical use |
|---|---|---|
| Per tensor | 1 | 8-bit activations |
| Per channel (row) | $d_{\text{out}}$ | 8-bit weights, standard |
| Per group of 64–128 | $n / g$ | 4-bit weights, standard |
Group size $g$ trades storage for fidelity: the scales themselves cost $16/g$ bits per weight when stored as fp16. At $g = 128$ that is 0.125 bits, so "4-bit" in practice means about 4.125 bits. Papers that omit this over-report their compression ratio.
Post-training versus quantisation-aware
Post-training quantisation (PTQ) needs no gradients. Weight-only PTQ needs no data at all; activation quantisation needs a small calibration set — typically 128 to 512 samples — to estimate ranges.
The two methods that define current practice for large models:
- GPTQ (Frantar et al., 2022) quantises weights column by column, using an approximation of the layer Hessian $H = 2XX^\top$ to update the not-yet-quantised weights so they compensate for the error already introduced. Cost is $O(d^3)$ per layer for the Cholesky, which is tractable per-layer and not globally.
- AWQ (Lin et al., 2023) observes that the salient weights are identified by activation magnitude rather than weight magnitude, and rescales channels before quantisation so that salient channels get effectively more resolution.
Quantisation-aware training (QAT) inserts fake-quantise nodes in the forward pass and trains through them. The rounding operator has zero gradient almost everywhere, so the straight-through estimator (Bengio, Léonard and Courville, 2013) is used: treat $\partial \hat{r} / \partial r = 1$ inside the clipping range and $0$ outside. QAT typically recovers most of the gap at 4 bits and below, at the cost of a full training run.
The practical decision rule: PTQ at 8 bits, PTQ with GPTQ or AWQ at 4 bits, QAT below 4 bits or when PTQ measurably fails.
Below four bits
- LLM.int8() (Dettmers et al., 2022) — mixed int8 with fp16 outlier decomposition, essentially lossless.
- QLoRA (Dettmers et al., 2023) — 4-bit NormalFloat, a data type whose levels are the quantiles of a normal distribution, plus double quantisation of the scales. See LoRA.
- BitNet b1.58 (Ma et al., 2024) — ternary weights in ${-1, 0, +1}$, about 1.58 bits, trained from scratch rather than converted. Reported parity with fp16 baselines at 3B parameters and above. Note the condition: trained from scratch. Post-training conversion to ternary still fails, exactly as the 2-bit row in the developer block fails.
Reading
- Jacob et al., Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference, CVPR 2018 — arxiv.org/abs/1712.05877
- Krishnamoorthi, Quantizing deep convolutional networks for efficient inference: a whitepaper, 2018 — arxiv.org/abs/1806.08342
- Dettmers et al., LLM.int8(), NeurIPS 2022 — arxiv.org/abs/2208.07339
- Frantar et al., GPTQ, ICLR 2023 — arxiv.org/abs/2210.17323
- Lin et al., AWQ, MLSys 2024 — arxiv.org/abs/2306.00978
- Ma et al., The Era of 1-bit LLMs: BitNet b1.58, 2024 — arxiv.org/abs/2402.17764