Relative position embeddings

Shaw et al. (2018)#

The first relative formulation adds learned vectors to keys and values, indexed by clipped relative distance:

$$ e_{ij} = \frac{(x_i W^Q)\big(x_j W^K + a^K_{ij}\big)^\top}{\sqrt{d_z}}, \qquad z_i = \sum_j \alpha_{ij}\big(x_j W^V + a^V_{ij}\big) $$

with $a^K_{ij} = w^K_{\mathrm{clip}(j-i,\,k)}$ and $\mathrm{clip}(x,k) = \max(-k, \min(k, x))$. The clip at $k$ (typically 8 to 16) bounds the table at $2k+1$ vectors of width $d_z$ per head.

Cost: $O(n^2 d)$ extra multiplies per head per layer for the key term, plus $(2k+1)d$ parameters. The value term $a^V$ was later found to contribute little and is usually dropped.

Transformer-XL#

Dai et al. (2019) expand the logit with sinusoidal relative encodings and two learned global bias vectors:

$$ A_{ij} = \underbrace{q_i^\top k_j}_{(a)} + \underbrace{q_i^\top W_{k,R}\, r_{i-j}}_{(b)} + \underbrace{u^\top k_j}{(c)} + \underbrace{v^\top W{k,R}\, r_{i-j}}_{(d)} $$

Term (a) is content-content. Term (b) is content-position. Term (c) is a learned global key bias, and (d) a learned global position bias; $u, v \in \mathbb{R}^{d}$ replace the query-dependent absolute terms of the naive expansion. Here $r_{i-j}$ is a fixed sinusoidal vector of the relative offset, so no table grows with length.

The implementation uses a shift trick to compute all $r_{i-j}$ terms in one matmul, reducing the cost from $O(n^2 d)$ to a single $O(n^2 d)$ product plus an $O(n^2)$ reindex.

T5#

Raffel et al. (2020) reduce the whole idea to a scalar added to the logit:

$$ A_{ij} = q_i^\top k_j + b^{(h)}_{\,\beta(j - i)} $$

with $\beta$ the bucketing function and $b \in \mathbb{R}^{B \times H}$ for $B$ buckets and $H$ heads. Default $B = 32$, $\max_\text{distance} = 128$. The bucketing is exact for the first $B/2$ (or $B/4$ bidirectional) offsets and logarithmic thereafter:

$$ \beta(\delta) = \begin{cases} |\delta| & |\delta| < B_{\text{exact}} \[4pt] B_{\text{exact}} + \left\lfloor \dfrac{\ln!\big(|\delta| / B_{\text{exact}}\big)}{\ln!\big(D_{\max} / B_{\text{exact}}\big)} \big(B - B_{\text{exact}}\big) \right\rfloor & \text{otherwise} \end{cases} $$

clamped to $B - 1$. Parameters: $BH$ per model, since the table is computed in layer zero and shared. For T5-base that is $32 \times 12 = 384$ numbers, against $512 \times 768 = 393{,}216$ for a BERT-style absolute table.

T5 also omits the $1/\sqrt{d_k}$ scaling and compensates in initialisation.

Placement in the taxonomy#

Setting $b^{(h)}_{\beta(\delta)} = -m_h |\delta|$ with fixed $m_h$ recovers ALiBi exactly. T5's bias is therefore the learned, bucketed generalisation of ALiBi, and ALiBi is T5's bias with the table replaced by a hand-chosen linear function. Their behaviour differs because a learned table can be non-monotone and head-specific in ways a single slope cannot.

Two-dimensional variants#

Liu et al. (2021), Swin Transformer, use a relative position bias inside each attention window. For a $M \times M$ window the offsets span $(2M-1)^2$ values, and the bias table is indexed by the flattened 2-D offset. This is a direct port of T5's scalar bias to a grid, and it is the standard positional mechanism in windowed vision transformers.

Empirical standing#

Kazemnejad et al. (2023) rank T5's relative bias first among explicit positional encodings for downstream length generalisation in decoder-only models, ahead of ALiBi and RoPE, with NoPE ahead of all of them on their benchmark. Press et al. (2022) report T5 bias extrapolates well but is markedly slower than ALiBi in wall-clock terms, because the bucket gather and the extra tensor are not free.

Speed is the reason relative bias lost to RoPE in large-scale decoders. RoPE folds into the query and key projections with elementwise operations and composes with fused attention kernels. A gathered bias tensor of shape $(H, n, n)$ does not, without kernel support.

Papers#

• Shaw, Uszkoreit and Vaswani, Self-Attention with Relative Position Representations, NAACL 2018 — arxiv.org/abs/1803.02155
• Dai et al., Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context, ACL 2019 — arxiv.org/abs/1901.02860
• Raffel et al., Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer (T5), JMLR 2020 — arxiv.org/abs/1910.10683
• Liu et al., Swin Transformer, ICCV 2021 — arxiv.org/abs/2103.14030

What to learn next#

• Models with no positional encoding at all — the case for removing the mechanism entirely.
• ALiBi — the same bias idea with a fixed function instead of a table.
• Attention — where the bias lands.