GPTQ

The layer-wise objective#

GPTQ (Frantar et al., 2023) does not minimise the weight error. It minimises the output error of each layer, independently:

$$ \hat W = \arg\min_{W'} \; \lVert W X - W' X \rVert_2^2 \quad \text{s.t. } W' \text{ quantised} $$

$X \in \mathbb{R}^{d_{\text{in}} \times N}$ holds $N$ calibration activations. The objective is quadratic in $W'$, with Hessian

$$ H = 2 X X^{\top} $$

which is why calibration data is required at all: $H$ is the calibration data, in second-moment form. Note $H$ is shared by every output row, so one Cholesky serves the whole matrix.

From OBS to GPTQ#

The lineage is Optimal Brain Surgeon (Hassibi and Stork, 1992) → OBQ (Frantar and Alistarh, 2022) → GPTQ.

OBQ greedily selects the weight whose quantisation costs least, quantises it, and applies the closed-form optimal update to the remaining unquantised weights:

$$ \delta_F = -\frac{w_q - \mathrm{quant}(w_q)}{[H_F^{-1}]_{qq}} \cdot \left(H_F^{-1}\right)_{:,q} $$

with $F$ the set of not-yet-quantised indices. This is exact, and it costs $O(d_{\text{row}} \cdot d_{\text{col}}^3)$ — days for a billion-parameter model.

GPTQ makes three changes that turn it into an hour:

1. Fixed order. Quantise columns left to right rather than greedily. Empirically the ordering barely matters for large layers, and it means every row uses the same $H^{-1}$, so it is computed once.
2. Lazy batch updates. Apply updates to a block of 128 columns at a time and only propagate to the rest of the matrix once per block. This turns a memory-bandwidth-bound inner loop into compute-bound matrix multiplies.
3. Cholesky reformulation. Repeated row/column removal from $H^{-1}$ is numerically unstable at scale. Precomputing the upper Cholesky factor of $H^{-1}$ gives exactly the required sequence of rows, stably.

Reported cost: OPT-175B and BLOOM-176B quantised to 3–4 bits in about four GPU-hours, with near-negligible perplexity increase — the first demonstration that 175B-class models could be quantised to 4 bits without retraining.

act_order / desc_act#

Processing columns in descending order of $\mathrm{diag}(H)$ — the most active input channels first — measurably improves quality, especially at 3 bits and on smaller models. The reason is that the compensation budget is largest early, so the channels with the most influence on the output get the best treatment.

Its cost is structural: with group-wise scales, reordering means group members are no longer contiguous, requiring an index permutation at inference. Some fast kernels (older Marlin paths) do not support that combination, so desc_act is a real quality-versus-throughput decision rather than a free win.

Where GPTQ is weak#

It is layer-wise and greedy. Each layer is optimised assuming the previous layers are unquantised, so errors compound across depth. Nagel et al., 2020 (AdaRound) and Li et al., 2021 (BRECQ) address this with block-wise reconstruction; Shao et al., 2024 (OmniQuant) learn clipping thresholds and equivalent transformations by gradient descent on a block objective, and beat GPTQ especially at 3 bits and below.

It overfits the calibration set at low bit widths. At 2–3 bits, results become noticeably sensitive to which calibration texts were used. Reporting a 3-bit result without stating the calibration set is not a reproducible claim.

It does not touch activations. GPTQ is a weight-only method. For W8A8 you need SmoothQuant or LLM.int8() as well.

GPTQ against AWQ#

They optimise different things and the comparison is genuinely close.

Published head-to-heads disagree by model and bit width. The practical advice is to run both on your own evaluation and pick, rather than to trust a table.

Kernels#

A GPTQ checkpoint is only fast if a matching kernel exists. The main families:

• ExLlamaV2 — fast, widely available, batch-1 oriented.
• Marlin — mixed-precision int4×fp16 kernels reaching near-ideal 4× speed-up at moderate batch sizes; needs Ampere or later and constrains group size and ordering.
• Triton / torch fallback — always works, often slower than fp16 at large batch, because dequantisation cost dominates when you are compute-bound rather than memory-bound.

That last point is the one people are surprised by: weight-only 4-bit quantisation speeds up small-batch decoding and can slow down large-batch prefill.

Papers#

• Hassibi and Stork, Second Order Derivatives for Network Pruning: Optimal Brain Surgeon, NeurIPS 1992.
• Nagel et al., Up or Down? Adaptive Rounding for Post-Training Quantization (AdaRound), ICML 2020 — arxiv.org/abs/2004.10568
• Li et al., BRECQ, ICLR 2021 — arxiv.org/abs/2102.05426
• Frantar and Alistarh, Optimal Brain Compression, NeurIPS 2022 — arxiv.org/abs/2208.11580
• Frantar et al., GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers, ICLR 2023 — arxiv.org/abs/2210.17323
• Shao et al., OmniQuant, ICLR 2024 — arxiv.org/abs/2308.13137
• Frantar et al., MARLIN: Mixed-precision Auto-Regressive Parallel Inference on Large Language Models, 2024 — arxiv.org/abs/2408.11743

What to learn next#

• AWQ — the other main 4-bit method, with a different idea.
• Measuring what quantisation costs you — how to compare the two honestly.
• GGUF and llama.cpp quantisation — the CPU-first family of formats.