Cross-attention

Definition#

With target states $Y \in \mathbb{R}^{T_y \times d}$ and encoder memory $Z \in \mathbb{R}^{T_z \times d_z}$:

$$ \operatorname{CrossAttn}(Y, Z) = \operatorname{softmax}!\left( \frac{(Y W_Q)(Z W_K)^\top}{\sqrt{d_k}} \right) Z W_V $$

The attention matrix is $T_y \times T_z$. Setting $Z = Y$ recovers self-attention exactly, so self-attention is the special case, not the general one.

Lineage#

Cross-attention is the original attention. Bahdanau et al. (2015), arXiv:1409.0473, introduced it to fix the fixed-length bottleneck in translation. Scoring used a small feedforward network. Luong et al. (2015), arXiv:1508.04025, replaced that with a dot product and catalogued global against local variants. Vaswani et al. (2017) kept cross-attention and added self-attention alongside it. The paper's title refers to removing recurrence, not to inventing attention.

Self-attention is the later idea. Textbook orderings that present it first invert the history.

Cost and caching#

Per decoding step $t$, per layer:

• Cross-attention: $O(T_z d)$. Constant in $t$, since the memory does not grow.
• Causal self-attention: $O(t d)$. Grows with what has been generated.

The encoder KV cache is written once, size $2 L T_z d_k$ bytes, and read every step. In a decoder-only model that content lives in the same growing cache as the generated tokens. The two designs converge in cost when $T_z \gg T_y$. The real difference is architectural separation, not arithmetic — a point made concrete in decoder-only vs encoder-decoder.

Alignment quality is not free#

Attention weights are frequently displayed as word alignments. Two papers together establish the careful position: Jain and Wallace (2019) and Wiegreffe and Pinter (2019). Attention weights are one of many distributions that produce the same output. They are evidence about information flow, not a faithful explanation of the decision. For translation, cross-attention correlates well with human alignments in some heads and poorly in others. The alignment-extraction literature selects or supervises specific heads rather than averaging them.

Cross-attention as a modality bridge#

Three designs dominate for connecting a non-text encoder to a language model:

• Interleaved cross-attention layers. Flamingo (Alayrac et al., 2022, arXiv:2204.14198) freezes both towers and inserts gated cross-attention layers. The gate is initialised to zero, so the language model is unchanged at step zero.
• Prefix projection. BLIP-2 (Li et al., 2023, arXiv:2301.12597) and the LLaVA family take a different route. Encoder features are projected into the token embedding space and prepended. The language model then uses ordinary self-attention. Simpler, and it consumes context length.
• Perceiver resampling. A fixed set of learned latent queries cross-attends into a variable-length input. The result is a constant-length summary (Jaegle et al., 2021, arXiv:2103.03206). This decouples cost from input length and is how many systems handle video.

The perceiver pattern is worth noting for a structural reason. The queries carry no information about the input at all. They are learned constants. Everything content-dependent arrives through the keys and values.

Papers#

• Bahdanau et al., Neural Machine Translation by Jointly Learning to Align and Translate, 2015 — arxiv.org/abs/1409.0473
• Luong et al., Effective Approaches to Attention-based NMT, 2015 — arxiv.org/abs/1508.04025
• Jaegle et al., Perceiver, 2021 — arxiv.org/abs/2103.03206
• Alayrac et al., Flamingo, 2022 — arxiv.org/abs/2204.14198
• Jain and Wallace, Attention is not Explanation, 2019 — arxiv.org/abs/1902.10186
• Wiegreffe and Pinter, Attention is not not Explanation, 2019 — arxiv.org/abs/1908.04626

What to learn next#

• Why attention costs grow with the square of length — where the bill comes from.
• Vision-language models — cross-attention joining pictures to words.
• Whisper — cross-attention joining sound to words.