Decoder-only vs encoder-decoder

Two ways to arrange the same block - one tower that continues a single stream of text, or two towers where one reads and the other writes.

By Pranay Mahendrakar
12 min read
3 reading levels
Updated 31 Aug 2026

Read these first

On this page 6

The two-tower version
The one-tower version
Why one tower won
Where two towers still make sense
Remember this
What to learn next

One lesson, three depths. Pick the one that fits you today — you can switch any time.

Beginner — No maths. Plain English.

There are two ways to arrange transformer blocks. Two towers, where one reads and the other writes. Or one tower, where reading and writing share a page.

Imagine you have received a letter and you need to reply. Two ways to do it.

Two sheets. Read the letter carefully, on its own page. Then take a fresh sheet and write your reply, glancing back at the letter whenever you need to. Two pages, two jobs.

One sheet. Write your reply directly underneath the letter, on the same page, continuing downward. The letter is now the earlier part of one long document.

Both work. The first is an encoder-decoder. The second is decoder-only, and it is what almost every chatbot you have used is built on.

The two-tower version

The encoder reads. Every word in the input can look at every other word, forwards and backwards. The input is already complete. Nothing is being predicted here, so there is nothing to peek at.

The decoder writes. It produces one word at a time, so it must not see its own future. It also glances back at the encoder's reading, which is what cross-attention does.

This was the original 2017 design, built for translation, where the two sides really are two different things.

The one-tower version

Put everything in one sequence. The question, the document, the instructions, and the answer all sit end to end.

Every position may look backwards and not forwards. The answer positions can see the question, because the question came earlier. The question positions cannot see the answer, and do not need to.

No second tower. No cross-attention. One kind of block, repeated.

   ENCODER-DECODER                    DECODER-ONLY

   [ the letter ]                     [ the letter | your reply ]
     reads freely, both ways            each word sees only what
          │                              came before it
          ▼
   [ your reply ]
     writes forward only,
     glancing back at the letter

Why one tower won

Three reasons, in order of how much they mattered.

It trains on anything. A decoder-only model needs nothing but a stream of text. Every position in every document is a free training example. The two-tower design needs paired data: this input goes with that output. Paired data is far rarer than plain text.

It is simpler. One kind of block, one mask, one stack. Less code, fewer things to tune, easier to make fast.

It turned out to be enough. The expectation was that separating reading from writing would help. In practice a single well-trained tower handles both. Any task can be phrased as "here is some text, continue it".

Where two towers still make sense

The two-tower design is not obsolete.

When the input is not text. Speech recognition reads sound and writes words. Image captioning reads pixels and writes words. Here the two sides really are different things, and giving them different towers is natural.

When the input is read many times. An encoder reads once. Its result can be reused for many different outputs. For a fixed document answered repeatedly, that is a real saving.

When you only need reading. A model that classifies or searches never writes anything. It wants an encoder with no mask at all, so every word sees every other word. That is what BERT is, and for search and classification it remains excellent.

Remember this

Encoder-decoder: one tower reads freely, another writes forward and glances back.
Decoder-only: one tower, one sequence, everything looks backwards only.
Decoder-only won for text because it trains on any text at all, with no pairing needed.

What to learn next

Counting a model's parameters by hand — turning any config file into an exact number.
BERT — the encoder-only family, in detail.
Whisper — a modern encoder-decoder that is not going anywhere.

Developer — Code and libraries.

Setup

bash

pip install torch

The three attention patterns, side by side

architectures.py

import torch

src = ["main", "chai", "peeta", "hoon"]
tgt = ["I", "drink", "tea"]

def show(mask, rows, cols, title):
    print(f"\n{title}   (. = allowed, X = blocked)")
    print("           " + "".join(f"{c:>8s}" for c in cols))
    for r, row in zip(rows, mask):
        print(f"{r:>9s}  " + "".join(f"{'X' if b else '.':>8s}" for b in row))

full = torch.zeros(len(src), len(src), dtype=torch.bool)
show(full, src, src, "encoder self-attention: every token reads every token")

causal = torch.triu(torch.ones(len(tgt), len(tgt), dtype=torch.bool), diagonal=1)
show(causal, tgt, tgt, "decoder self-attention: nobody reads the future")

cross = torch.zeros(len(tgt), len(src), dtype=torch.bool)
show(cross, tgt, src, "cross-attention: target rows, source columns, no mask at all")

print("\nshapes tell you which is which:")
print(f"  encoder self : {len(src)} x {len(src)}  square, source only")
print(f"  decoder self : {len(tgt)} x {len(tgt)}  square, target only, triangular")
print(f"  cross        : {len(tgt)} x {len(src)}  rectangular, joins the two")

print("\nsame job, two shapes of model. what changes:")
d, L = 1024, 12
enc_dec = 2 * L * (4*d*d) + L * (4*d*d) + 2 * L * (8*d*d)   # enc self + dec self + cross + FFNs
dec_only = 2 * L * (4*d*d) + 2 * L * (8*d*d)                # 24 blocks, no cross-attention
print(f"  encoder-decoder, {L} layers each side: {enc_dec:,} weight parameters")
print(f"  decoder-only, {2*L} layers          : {dec_only:,} weight parameters")
print("  -> the encoder-decoder pays extra for a whole third attention module per layer")

print("\nre-reading a long prompt, per new output token:")
prompt, so_far = 2000, 50
print(f"  encoder-decoder: {prompt} source keys cached once, "
      f"{so_far} target keys cached -> {prompt + so_far} keys read")
print(f"  decoder-only   : prompt and output live in ONE sequence -> "
      f"{prompt + so_far} keys read")
print("  -> identical work. the encoder buys separation of concerns, not speed.")

Output


encoder self-attention: every token reads every token   (. = allowed, X = blocked)
               main    chai   peeta    hoon
     main         .       .       .       .
     chai         .       .       .       .
    peeta         .       .       .       .
     hoon         .       .       .       .

decoder self-attention: nobody reads the future   (. = allowed, X = blocked)
                  I   drink     tea
        I         .       X       X
    drink         .       .       X
      tea         .       .       .

cross-attention: target rows, source columns, no mask at all   (. = allowed, X = blocked)
               main    chai   peeta    hoon
        I         .       .       .       .
    drink         .       .       .       .
      tea         .       .       .       .

shapes tell you which is which:
  encoder self : 4 x 4  square, source only
  decoder self : 3 x 3  square, target only, triangular
  cross        : 3 x 4  rectangular, joins the two

same job, two shapes of model. what changes:
  encoder-decoder, 12 layers each side: 352,321,536 weight parameters
  decoder-only, 24 layers          : 301,989,888 weight parameters
  -> the encoder-decoder pays extra for a whole third attention module per layer

re-reading a long prompt, per new output token:
  encoder-decoder: 2000 source keys cached once, 50 target keys cached -> 2050 keys read
  decoder-only   : prompt and output live in ONE sequence -> 2050 keys read
  -> identical work. the encoder buys separation of concerns, not speed.

The last line is the one that surprises people

Encoder-decoder is frequently described as cheaper for long inputs, because the encoder runs once. The count says otherwise.

A decoder-only model also encodes the prompt exactly once, during the prefill pass. It caches the keys and values. From then on it reads the same number of cached keys per step. The two designs do the same work.

What differs is representational. The encoder reads the input bidirectionally, so every input token has seen the whole input. In a decoder-only model, input token five has only seen tokens one to five. That is a genuine asymmetry, and it is the strongest technical argument for keeping an encoder.

The three families, and what each is for

Family	Mask	Trained by	Good at	Example
Encoder-only	none	masked-token prediction	classifying, embedding, search	BERT
Decoder-only	causal	next-token prediction	generating anything	GPT, Llama
Encoder-decoder	both	corrupted-span reconstruction	fixed input to fixed output	T5, Whisper

Encoder-only models are not obsolete and are frequently the right answer. For semantic search or a text classifier, an encoder is smaller and faster. It is often more accurate than a generative model on the same job. See BERT.

Building a decoder-only stack

python

import torch, torch.nn as nn, torch.nn.functional as F

class Decoder(nn.Module):
    def __init__(self, V, d, L, n_heads, max_T):
        super().__init__()
        self.tok, self.pos = nn.Embedding(V, d), nn.Embedding(max_T, d)
        layer = nn.TransformerEncoderLayer(d, n_heads, 4 * d, batch_first=True,
                                           norm_first=True)      # 'Encoder' is a misnomer here
        self.blocks = nn.TransformerEncoder(layer, L, enable_nested_tensor=False)
        self.norm, self.head = nn.LayerNorm(d), nn.Linear(d, V, bias=False)
    def forward(self, idx):
        T = idx.size(1)
        h = self.tok(idx) + self.pos(torch.arange(T))
        mask = nn.Transformer.generate_square_subsequent_mask(T)  # the causal mask
        return self.head(self.norm(self.blocks(h, mask=mask, is_causal=True)))

m = Decoder(V=100, d=32, L=2, n_heads=4, max_T=16)
print(m(torch.randint(0, 100, (2, 10))).shape)

Output

torch.Size([2, 10, 100])

enable_nested_tensor=False is there to silence a warning. The nested-tensor fast path is incompatible with norm_first=True, and PyTorch says so on every construction.

Note the naming trap. nn.TransformerEncoderLayer is what you use for a decoder-only model. PyTorch's TransformerDecoderLayer includes cross-attention and expects an encoder memory tensor, which a decoder-only model does not have. The names describe the 2017 paper's roles, not the modern usage.

Common mistakes

Reaching for nn.TransformerDecoderLayer to build a GPT. It wants a memory argument. Use TransformerEncoderLayer with a causal mask, as above.

Using a decoder-only model for embeddings without adapting it. The final hidden state of a causal model has only seen the text to its left. For sentence embeddings that is a real handicap, and specialised recipes exist to work around it.

Masking an encoder. If the model is classifying or embedding, it should see the whole input. Applying a causal mask halves the information available for no reason.

Assuming a prefix-language-model mask is exotic. Allow bidirectional attention over the prompt and causal attention over the continuation. A single stack then gets most of the encoder's benefit. It is the design used by UniLM and by T5's prefix-LM variant.

Try it yourself

Build the prefix mask. Take a sequence with a prompt of length p and a continuation. Allow all attention among the first p positions, and causal attention after. Print it. Confirm it is a square with a full block in the top-left and a triangle below. Then reason about why it breaks the KV cache guarantee if you extend the prompt afterwards.

What to learn next

Counting a model's parameters by hand — turning any config file into an exact number.
BERT — the encoder-only family, in detail.
Whisper — a modern encoder-decoder that is not going anywhere.

Researcher — Mathematics and papers.

The three masks

Let $M \in {0, -\infty}^{T \times T}$ be the additive attention mask.

Encoder-only: $M_{ij} = 0$ for all $i, j$. Bidirectional.
Decoder-only: $M_{ij} = -\infty$ for $j > i$. Autoregressive.
Prefix-LM: $M_{ij} = 0$ for $j \le p$ or $j \le i$; $-\infty$ otherwise, with $p$ the prompt length.

The encoder-decoder is a decoder with the prefix carried by a separate stack, joined by cross-attention. The alternative is a block-structured mask in one stack. Raffel et al. (2020), arXiv:1910.10683, make exactly this observation and compare the three empirically in their Section 3.2.

Parameter and cost accounting

Per layer, ignoring norms:

$$ P_{\text{enc}} = 12d^2, \qquad P_{\text{dec}} = 12d^2 + 4d^2 = 16d^2 $$

The decoder's extra $4d^2$ is cross-attention. An encoder-decoder with $L$ layers each side holds $28 L d^2$; a decoder-only model of $2L$ layers holds $24 L d^2$. At matched parameters, the decoder-only model gets more layers.

Inference cost per generated token, with source length $S$ and generated length $t$:

Encoder-decoder: cross-attention $O(L S d)$, self-attention $O(L t d)$.
Decoder-only: self-attention over the combined cache, $O(L (S + t) d)$.

Asymptotically equal. The encoder-decoder's KV cache splits into a fixed part and a growing part, which can help memory management. Its cross-attention cache is identical across a beam, a real saving in beam search.

The empirical comparisons that exist

Raffel et al. (2020), T5, compare all three at matched parameters and matched compute. The tasks are transfer benchmarks. Encoder-decoder wins at matched parameters, and the gap narrows at matched compute. An encoder-decoder with $L$ layers each side runs each parameter over fewer tokens than a $2L$-layer decoder. The two matchings are genuinely different comparisons.

Wang et al. (2022), arXiv:2204.05832, is the most careful large-scale study. Three findings. Causal decoder-only with autoregressive pretraining gives the strongest zero-shot generalisation immediately after pretraining. Encoder-decoder with span corruption is strongest after multitask finetuning. Architectures can be adapted between the two regimes cheaply.

Tay et al. (2023), UL2, arXiv:2205.05131, propose a mixture of denoising objectives to close the gap. Objective and architecture are entangled. Comparisons of one holding the other fixed are of limited value.

The honest summary is this. Decoder-only dominance is as much about the pretraining objective, data availability and engineering simplicity as about the architecture. The controlled evidence does not show encoder-decoder to be worse.

Where encoder-decoder persists

Speech recognition. Whisper (Radford et al., 2022, arXiv:2212.04356) encodes a log-mel spectrogram and decodes text. The two modalities have different lengths, different rates and different natural representations.
Modality bridging generally. A frozen non-text encoder plus cross-attention into a language model is one standard vision-language design. See vision-language models.
Fixed-input, many-output workloads, where the encoder result is genuinely reused.

Encoder-only models remain the efficient answer for retrieval and classification. A bidirectional representation and a small parameter count both matter there.

Papers

Vaswani et al., Attention Is All You Need, 2017 — arxiv.org/abs/1706.03762
Devlin et al., BERT, 2019 — arxiv.org/abs/1810.04805
Raffel et al., Exploring the Limits of Transfer Learning (T5), 2020 — arxiv.org/abs/1910.10683
Wang et al., 2022. What Language Model Architecture and Pretraining Objective Work Best for Zero-Shot Generalization? — arxiv.org/abs/2204.05832
Tay et al., UL2: Unifying Language Learning Paradigms, 2023 — arxiv.org/abs/2205.05131

What to learn next

Counting a model's parameters by hand — turning any config file into an exact number.
BERT — the encoder-only family, in detail.
Whisper — a modern encoder-decoder that is not going anywhere.

🟢 Beginner — No maths. Plain English.

The two-tower version#

The one-tower version#

Why one tower won#

Where two towers still make sense#

Remember this#

What to learn next#

🟡 Developer — Code and libraries.

Setup#

The three attention patterns, side by side#

The last line is the one that surprises people#

The three families, and what each is for#

Building a decoder-only stack#

Common mistakes#

Try it yourself#

What to learn next#

🔴 Researcher — Mathematics and papers.

The three masks#

Parameter and cost accounting#

The empirical comparisons that exist#

Where encoder-decoder persists#

Papers#

What to learn next#

Beginner — No maths. Plain English.

The two-tower version

The one-tower version

Why one tower won

Where two towers still make sense

Remember this

What to learn next

Developer — Code and libraries.

Setup

The three attention patterns, side by side

The last line is the one that surprises people

The three families, and what each is for

Building a decoder-only stack

Common mistakes

Try it yourself

What to learn next

Researcher — Mathematics and papers.

The three masks

Parameter and cost accounting

The empirical comparisons that exist

Where encoder-decoder persists

Papers

What to learn next