Stretching RoPE for longer context

The failure being repaired#

For RoPE with base $b$ and head dimension $d$, pair $i$ has angular frequency $\theta_i = b^{-2i/d}$ and wavelength $\lambda_i = 2\pi b^{2i/d}$. Define the number of full rotations completed within training context $L$:

$$ r(i) = \frac{L}{\lambda_i} $$

Pairs with $r(i) \gg 1$ are fully exercised during training. Pairs with $r(i) < 1$ present, at inference beyond $L$, phases outside the training support. Direct extrapolation therefore fails not because RoPE is undefined past $L$ — it is perfectly well defined — but because the low-frequency subspace is out of distribution.

Position Interpolation#

Chen et al. (2023) rescale positions rather than extrapolate:

$$ f'(x, m) = f!\left(x, \frac{mL}{L'}\right), \qquad s = \frac{L'}{L} $$

Equivalently $\theta_i \mapsto \theta_i / s$, so every wavelength is multiplied by $s$. All phases remain inside the trained range. The paper extends Llama to 32k with 1000 finetuning steps.

The cost is uniform. High-frequency pairs, which encode local ordering, are compressed by $s$ as well, so adjacent tokens become $s$ times closer in phase. This crowds exactly the signal the model relies on for local syntax.

NTK-aware scaling#

Proposed by the pseudonymous bloc97 (2023) on the LocalLLaMA forum and analysed in the YaRN paper. Instead of scaling positions, scale the base:

$$ b' = b \cdot s^{\frac{d}{d-2}} $$

This choice makes the lowest-frequency pair receive scaling $s$ while the highest-frequency pair is left essentially untouched, with a smooth geometric progression in between. It does not require finetuning to give usable results, which is why it appeared first in community deployments.

The "dynamic" variant recomputes $b'$ from the current sequence length, using $s = \max(1, \ell / L)$, so short sequences run with the original base. That is exactly what the developer output shows: identical wavelengths at $\ell = 2048$, changed wavelengths at $\ell = 16384$.

YaRN#

Peng, Quesnelle, Fan and Shippole (2023) combine two ideas.

NTK-by-parts interpolation. Define the ratio

$$ r(i) = \frac{L}{\lambda_i} = \frac{L}{2\pi b^{2i/d}} $$

and a ramp

$$ \gamma(r) = \begin{cases} 0 & r < \alpha \ 1 & r > \beta \ \dfrac{r - \alpha}{\beta - \alpha} & \text{otherwise} \end{cases} $$

Then interpolate per pair:

$$ h(\theta_i) = \big(1 - \gamma(r(i))\big)\frac{\theta_i}{s} + \gamma(r(i))\,\theta_i $$

Pairs completing many rotations ($r > \beta$) keep $\theta_i$ unchanged. Pairs completing fewer than $\alpha$ rotations get full interpolation $\theta_i / s$. Between them the ramp blends linearly. For Llama models the paper recommends $\alpha = 1$, $\beta = 32$; these are beta_slow and beta_fast in the transformers config, defaulting to 1 and 32.

This is directly visible in the developer output: three wavelengths equal to the default, two equal to the fully interpolated values, three intermediate.

Attention temperature. Divide the pre-softmax logits by $t$, with the empirical fit

$$ \sqrt{1/t} = 0.1 \ln(s) + 1 $$

At $s = 8$ this gives $1.2079$, matching the printed attention scale. The motivation is entropy: with $s$ times more tokens competing in one softmax, attention flattens, and a temperature restores sharpness. It costs nothing at runtime because it can be folded into the precomputed $\cos$ and $\sin$ tables.

The paper reports YaRN reaching the same perplexity as prior methods with roughly 10 times fewer tokens and 2.5 times fewer training steps.

Llama 3's rule#

Llama 3.1 uses a frequency-cutoff scheme, exposed as rope_type: "llama3" with low_freq_factor and high_freq_factor. Wavelengths shorter than a high-frequency threshold are untouched, those longer than a low-frequency threshold are divided by factor, and the band between is smoothly interpolated. Structurally this is the same idea as NTK-by-parts, parameterised by wavelength thresholds rather than by rotation counts. It was applied during pretraining, alongside a base of 500000, rather than as a post-hoc extension.

LongRoPE#

Ding et al. (2024) search for a per-dimension rescaling vector with an evolutionary algorithm rather than deriving it, and use a two-stage schedule to reach 2M tokens. Transformers exposes it as rope_type: "longrope" with explicit short_factor and long_factor lists, one entry per pair. It is used in the Phi model family.

Evaluation honestly#

Perplexity on long documents is the standard headline and is a weak proxy. Retrieval-style probes such as needle-in-a-haystack measure a narrow skill. Hsieh et al. (2024), RULER, construct tasks with controllable difficulty and find that models advertising 128k or more frequently sustain far shorter effective context, with several dropping below their claimed length by a wide margin. Treat an advertised context length as an input limit, not a capability claim.

Papers#

• Chen et al., Extending Context Window of Large Language Models via Position Interpolation, 2023 — arxiv.org/abs/2306.15595
• Peng et al., YaRN: Efficient Context Window Extension of Large Language Models, 2023 — arxiv.org/abs/2309.00071
• Ding et al., LongRoPE, 2024 — arxiv.org/abs/2402.13753
• Hsieh et al., RULER: What's the Real Context Size of Your Long-Context Language Models?, 2024 — arxiv.org/abs/2404.06654
• Hugging Face rotary embedding utilities — huggingface.co/docs/transformers/internal/rope_utils

What to learn next#

• What breaks when you exceed the trained length — measured, not asserted.
• Context window — what the number on a model card actually means.
• ALiBi — a design that needs no scaling at all.