Longformer and long-context encoders

The attention pattern, formally#

Longformer (Beltagy, Peters & Cohan, 2020) combines two attention patterns within the same model:

Local (sliding window) attention:
  token i attends to tokens { i - w/2, ..., i + w/2 }    for window size w

Global attention (a small, task-chosen set G of positions):
  token i in G attends to ALL n tokens, and is attended to by all n tokens

• w is the attention window size, configurable per layer (the checkpoint inspected in the developer block uses w = 512 at every layer).
• G is typically small relative to n — the [CLS]/<s> token by default, plus task-specific positions (all question tokens, for extractive QA) chosen based on which task Longformer is fine-tuned for.

Complexity for local attention alone is O(n * w), linear in sequence length n for fixed window w — a direct improvement over full attention's O(n^2). Global attention adds O(|G| * n), which stays small in practice because |G| is kept small by design; total complexity is O(n * w + |G| * n) = O(n * (w + |G|)), still linear in n.

Why global attention is necessary, not optional#

Pure local attention with no global component has a real limitation: information can only propagate w/2 positions per layer, since that is the furthest any single attention operation reaches. A fact at token 100 and a fact at token 4,000 can only interact after enough layers for local attention windows to chain together across the full distance — at 12 layers and w = 512, the effective receptive field is on the order of 12 * 256 = 3,072 positions, which may or may not cover a 4,000-token gap depending on layer count and window size. Global attention tokens sidestep this entirely: any token marked global reaches every other token in a single layer, and every other token reaches it in a single layer too, giving the network at least one direct, single-hop path for connecting distant information, regardless of local window size or layer depth.

Longformer's initialisation strategy#

Rather than pretraining from scratch — expensive, and unnecessary if most of the network's learned behaviour transfers — Longformer initialises its weights from a pretrained RoBERTa checkpoint. Its local attention window uses the same attention mechanism RoBERTa already learned; only the position embedding table needs extending, from RoBERTa's 512 positions to 4,096, done by copying the existing 512-position embeddings repeatedly to fill the extended table, then continuing MLM pretraining briefly on long documents to let the model adapt the newly-extended positions. This is a considerably cheaper path to a long-context model than pretraining an equivalent model from random initialisation, and the same "initialise from a shorter-context checkpoint, then extend" strategy reappears in later long-context adaptation work across the field, including several LLM context-extension techniques.

Comparison with other sparse-attention long-context designs#

Sparse Transformers (Child et al., 2019) and BigBird (Zaheer et al., 2020) combine similar local-window attention with different supplementary patterns — BigBird adds both global tokens (as Longformer does) and a set of random attention connections, arguing the combination provides a stronger theoretical guarantee of information flow across the whole sequence than local-plus-global alone. Empirically, the gap between these closely related sparse-attention designs on downstream benchmarks tends to be small relative to the gap between any of them and full attention on short sequences — the family shares its core trade-off, and differs mainly in secondary design choices layered on top of it.

Key references#

• Beltagy, I., Peters, M. & Cohan, A. (2020). Longformer: The Long-Document Transformer. arXiv:2004.05150
• Zaheer, M. et al. (2020). Big Bird: Transformers for Longer Sequences. arXiv:2007.14062
• Child, R. et al. (2019). Generating Long Sequences with Sparse Transformers. arXiv:1904.10509
• Liu, Y. et al. (2019). RoBERTa: A Robustly Optimized BERT Pretraining Approach. arXiv:1907.11692 — the checkpoint Longformer initialises from.

Current state and open problems#

Sparse and windowed attention, as a family, has been substantially displaced at the very largest scale by exact, hardware-optimised full attention — FlashAttention made full O(n^2) attention fast enough in practice, via better memory access patterns rather than a smaller asymptotic complexity, that many current long-context models use full attention directly at large context lengths instead of approximating it away. Longformer's specific approach nonetheless remains a genuinely useful, well-understood technique, and its local-plus-global attention pattern reappears as a component inside newer architectures — including ModernBERT's alternating local/global layer design, which borrows the same core idea (most layers local, some layers global) while combining it with rotary position embeddings rather than Longformer's extended absolute position table. Whether sparse attention patterns like Longformer's or exact-but-optimised full attention will dominate long-context model design longer term remains genuinely unsettled, with both approaches actively used across different current architectures.

What to learn next#

• ModernBERT — a current architecture combining a similar local/global pattern with rotary position embeddings instead.
• Fast attention and long context — the fuller landscape of techniques, including FlashAttention, this lesson's approach sits alongside.
• The 512-token wall — the problem this entire lesson exists to solve.