Cross-entropy and perplexity

Definition#

For a tokenised sequence $x_{1:T}$, perplexity is the exponentiated mean negative log-likelihood:

$$ \mathrm{PPL}(x_{1:T}) = \exp!\left(-\frac{1}{T}\sum_{t=1}^{T}\log p_\theta(x_t \mid x_{<t})\right) $$

$T$ is the token count, $p_\theta$ the model's conditional distribution. Since the exponent is exactly the cross-entropy loss $\mathcal{L}$ in nats, $\mathrm{PPL} = e^{\mathcal{L}}$.

Equivalently, perplexity is the reciprocal geometric mean of the assigned token probabilities:

$$ \mathrm{PPL} = \left(\prod_{t=1}^{T} p_\theta(x_t \mid x_{<t})\right)^{-1/T} $$

The geometric mean is why a single near-zero probability dominates the score, as the toy example above demonstrated.

Relation to entropy and cross-entropy#

Under the true distribution $q$:

$$ \mathbb{E}q[\mathcal{L}] = H(q, p\theta) = H(q) + D_{\mathrm{KL}}(q | p_\theta) $$

So $\mathrm{PPL} \geq e^{H(q)}$, with equality only for a perfect model. Perplexity has a floor set by the data, not by the model. Reported perplexity below the entropy of the test set is evidence of contamination, not of a better model.

Bits per byte#

Perplexity is not comparable across tokenisers. Bits per byte is:

$$ \mathrm{BPB} = \frac{\mathcal{L}{\text{total, nats}}}{\ln(2)\cdot n{\text{bytes}}} $$

$\mathcal{L}{\text{total, nats}}$ is the summed negative log-likelihood over the document, and $n{\text{bytes}}$ its length in UTF-8 bytes. Because the denominator is a property of the text rather than of the tokeniser, BPB compares any two models on the same corpus. The Pile (Gao et al., 2020) popularised it for exactly this reason, and it is the correct metric whenever vocabulary sizes differ.

The conversion, when the tokeniser is held fixed, is

$$ \mathrm{BPB} = \frac{\log_2 \mathrm{PPL} \cdot n_{\text{tokens}}}{n_{\text{bytes}}} $$

The strided-window protocol#

A model with context $C$ cannot score a document longer than $C$ in one pass. Three protocols exist and they do not agree:

1. Disjoint chunks. Split into non-overlapping blocks of $C$. Cheap, and pessimistic — every block's first tokens are scored with almost no context.
2. Fully sliding window, stride 1. Score each token with the maximum available context. Optimal, and costs one forward pass per token.
3. Strided window, stride $s < C$. Advance by $s$, score only the final $s$ tokens of each window. The standard compromise; the HuggingFace perplexity guide uses it.

Reported perplexity varies by tens of percent across these. Any comparison must state the stride. This is the largest single source of irreproducible perplexity numbers in the literature.

What perplexity does and does not predict#

Perplexity correlates strongly with downstream capability within an architecture family trained on similar data, which is why it anchors scaling laws. It correlates far more weakly across families.

Three documented failure modes:

• Post-training breaks the link. Instruction tuning and RLHF typically raise perplexity on generic web text while sharply improving usefulness. Perplexity is a pretraining metric; it is close to useless for judging an aligned chat model.
• Emergence is invisible. Wei et al., 2022 reported capabilities appearing abruptly with scale while loss fell smoothly. Schaeffer et al., 2023 (Are Emergent Abilities of Large Language Models a Mirage?) argued convincingly that the abruptness is an artefact of discontinuous metrics such as exact match, not of the underlying continuous improvement. Both papers agree perplexity alone will not show you the transition.
• Quantisation hides damage. A 4-bit model can move perplexity by under 1% while losing several points on reasoning benchmarks. See measuring what quantisation costs you.

Related measures#

Perplexity is a proper scoring rule in the strict sense: it is uniquely minimised by reporting the true conditional distribution. That is what token accuracy loses — see proper scoring rules.

Papers and sources#

• Jelinek et al., Perplexity — a measure of the difficulty of speech recognition tasks, 1977 — the original definition.
• Brown et al., An Estimate of an Upper Bound for the Entropy of English, 1992 — Computational Linguistics 18(1).
• Gao et al., The Pile, 2020 — arxiv.org/abs/2101.00027
• Wei et al., Emergent Abilities of Large Language Models, 2022 — arxiv.org/abs/2206.07682
• Schaeffer et al., Are Emergent Abilities of Large Language Models a Mirage?, 2023 — arxiv.org/abs/2304.15004
• HuggingFace, Perplexity of fixed-length models — huggingface.co/docs/transformers/perplexity

What to learn next#

• Packing documents into fixed-length batches — the batching that these losses are averaged over.
• Reading a loss curve — what the shape of the curve is telling you.
• Measuring what quantisation costs you — where perplexity misleads badly.