Pre-norm vs post-norm
Normalising before a sublayer leaves the residual path untouched; normalising after it rescales the whole running total every layer - and that one choice decides whether a deep model trains easily.
- 12 min read
- 3 reading levels
- Updated
Read these first
On this page 5
One lesson, three depths. Pick the one that fits you today — you can switch any time.
Beginner — No maths. Plain English.
The normalisation step can go before a layer's work or after it. That single choice changes how easy the model is to train.
Picture a bucket you are filling with water, mug by mug. There are two ways to keep things tidy.
Way one: measure each mug to a standard level before pouring it in. The bucket fills up steadily, and whatever was already in it is never touched.
Way two: pour the mug in, then tip the bucket out until it is back at a fixed level. Every pour resets the whole bucket, including everything that went in earlier.
The first way is pre-norm: normalise the input to a layer. The second is post-norm: normalise the result after adding.
Why the difference matters so much
Recall that a transformer's running total is a bucket that every layer pours into. That total carries the token's information forward.
With pre-norm, nothing ever disturbs the bucket. What layer one poured in is still exactly there at layer forty. There is a clean, untouched path from the beginning of the model to the end.
With post-norm, every layer rescales the entire bucket. After forty layers, the original contribution has been rescaled forty times over. There is no untouched path at all.
That untouched path is what a deep model needs to learn. Without it, the signal that tells early layers how to improve must survive forty rescalings. It arrives faint and unreliable.
The history, which explains a lot of confusing code
The 2017 paper that introduced transformers used post-norm. It worked, and it is still what you will find in older codebases.
But it needed careful handling. Training had to begin with a warm-up. For hundreds or thousands of steps, the learning rate starts near zero and climbs. Skip the warm-up with post-norm and the model often falls apart in the first few steps.
Around 2019 and 2020, several groups moved the normalisation before each sublayer. That removed the need for warm-up and made deep models far more forgiving. Almost everything trained since has used pre-norm.
You will meet two transformer implementations that look identical except for one line. One has a warm-up schedule and the other does not. This is why.
POST-NORM (2017) PRE-NORM (now standard)
x ──┬──────────────┐ x ──┬──────────────┐
│ │ │ │
│ sublayer normalise │
│ │ │ │
└──── add ◄────┘ sublayer │
│ │ │
normalise └─── add ◄────┘
│ │
out out
the whole total is the total is never
rescaled every layer touched by normalisationThe honest part
Post-norm is not worse across the board. When it trains successfully, it sometimes reaches a slightly better final quality. There are careful recipes that make very deep post-norm models work.
The trade is real. Pre-norm is easier and more forgiving. Post-norm is fussier, with a possible small upside. For anyone who is not running a large research team, pre-norm is the sane default.
There is also a middle option. The normalisation goes both before the sublayer and on its output before the addition. Several recent models use it. The design space here is genuinely still moving.
Remember this
- Pre-norm normalises the input to a layer, leaving the running total untouched.
- Post-norm normalises the running total itself, once per layer.
- Pre-norm trains deep models with much less fuss, which is why it became the default.
What to learn next
- RMSNorm — the normalisation that replaced LayerNorm in most current models.
- The residual stream — the path this decision either protects or interrupts.
- Gradient clipping — the other standard defence against a diverging run.
Developer — Code and libraries.
Setup
pip install torchBoth designs, same weights, same seed
import time
import torch
import torch.nn as nn
import torch.nn.functional as F
V, d, T, L = 24, 48, 12, 24
class Block(nn.Module):
def __init__(self, pre):
super().__init__()
self.pre = pre
self.norm = nn.LayerNorm(d)
self.fc1, self.fc2 = nn.Linear(d, 2 * d), nn.Linear(2 * d, d)
def branch(self, h):
return self.fc2(F.gelu(self.fc1(h)))
def forward(self, x):
if self.pre:
return x + self.branch(self.norm(x)) # pre-norm: the '+' path is untouched
return self.norm(x + self.branch(x)) # post-norm: the '+' path gets rescaled
def build(pre):
torch.manual_seed(0) # same starting weights both ways
return nn.Sequential(nn.Embedding(V, d),
*[Block(pre) for _ in range(L)],
nn.Linear(d, V))
print("what the residual stream does over 24 blocks:")
for pre in (True, False):
net = build(pre)
h = net[0](torch.randint(0, V, (2, T), generator=torch.Generator().manual_seed(1)))
sizes = []
with torch.no_grad():
for blk in net[1:-1]:
h = blk(h)
sizes.append(h.norm().item())
label = "pre-norm " if pre else "post-norm"
print(f" {label} block 0: {sizes[0]:7.2f} block 11: {sizes[11]:7.2f} "
f"block 23: {sizes[-1]:7.2f}")
print(" -> post-norm pins the stream to a fixed size at every single block.")
def train(pre, steps=301, lr=1e-3):
net = build(pre)
opt = torch.optim.Adam(net.parameters(), lr=lr) # no warmup, on purpose
g = torch.Generator().manual_seed(1)
out = []
for step in range(steps):
x = torch.randint(0, V, (16, T), generator=g)
loss = F.cross_entropy(net(x).reshape(-1, V), x.reshape(-1)) # task: copy the input
opt.zero_grad(); loss.backward(); opt.step()
if step % 100 == 0:
out.append((step, loss.item()))
return out
print("\ntraining both to copy their own input through all 24 blocks")
print("(takes about a minute on a laptop CPU)")
t0 = time.time()
for pre in (True, False):
hist = train(pre)
label = "pre-norm " if pre else "post-norm"
print(f" {label} " + " ".join(f"step {s:>3}: {l:.4f}" for s, l in hist))
print(f" elapsed: {time.time() - t0:.0f} seconds on the machine this was written on")what the residual stream does over 24 blocks: pre-norm block 0: 35.29 block 11: 43.97 block 23: 50.82 post-norm block 0: 33.94 block 11: 33.94 block 23: 33.94 -> post-norm pins the stream to a fixed size at every single block. training both to copy their own input through all 24 blocks (takes about a minute on a laptop CPU) pre-norm step 0: 3.5975 step 100: 0.0005 step 200: 0.0003 step 300: 0.0002 post-norm step 0: 3.4074 step 100: 0.0495 step 200: 0.0175 step 300: 0.0091 elapsed: 30 seconds on the machine this was written on
The elapsed time depends on your CPU. The loss values are reproducible on the same PyTorch build (2.5.1 here). They will drift slightly on another version or platform.
The two results
Post-norm's stream norm is 33.94 at block 0, block 11 and block 23. Not approximately — the same number three times. The final normalisation of each block pins the output to a fixed radius, so the stream cannot accumulate scale. Pre-norm's grows 35.29 to 50.82, roughly as the square root of depth. That is what a sum of near-independent contributions should do.
That pinning is precisely what removes the identity path. In pre-norm the block computes x + f(norm(x)). So x passes through with a derivative of one. In post-norm it computes norm(x + f(x)), and the normalisation sits between x and the output.
The training task is deliberately trivial: copy the input. All the model has to do is carry information through 24 blocks unharmed. Pre-norm reaches 0.0002. Post-norm plateaus at 0.0091, roughly 45 times higher, on the same weights, seed and learning rate.
Be careful with how strongly you read this. Post-norm did learn the task, and at this depth the gap is a nuisance rather than a failure. At 100 layers and a real learning rate without warm-up, post-norm frequently does not converge at all. This experiment shows the direction of the effect on hardware you have; it understates the size at scale.
The variants you will meet in real code
| Name | Formula | Seen in |
|---|---|---|
| Post-norm | norm(x + f(x)) | original transformer, BERT |
| Pre-norm | x + f(norm(x)) | GPT-2 onwards, Llama, most current models |
| Sandwich / peri-norm | x + norm2(f(norm1(x))) | Gemma 2, several 2024-onward models |
| DeepNorm | norm(alpha * x + f(x)) | very deep encoder-decoder stacks |
The sandwich variant is the interesting middle. The residual path stays clean like pre-norm. The branch output is bounded before it is added, capping how much any one layer can write.
Practical guidance
Use pre-norm unless you have a specific reason. It is what nearly every current open model uses and what nearly every tutorial assumes.
Pre-norm still needs a final normalisation. Because nothing rescales the stream, the last block's output can be large. Every pre-norm model ends with one more norm before the output head. Forgetting it is a real bug that produces oddly confident garbage.
Warm-up is still useful with pre-norm, for a different reason. It helps Adam build reliable second-moment estimates in the first few hundred steps. It is no longer load-bearing for stability.
If you must use post-norm, use warm-up, and consider the DeepNorm scaling if the model is deep. Do not port a post-norm architecture to a no-warm-up schedule and expect it to survive.
Common mistakes
Forgetting the final norm in a pre-norm model. The most common bug when writing a GPT from scratch. Symptoms: training works, generation is oddly deterministic or saturated.
Applying the norm to x in place. x = self.norm(x) followed by x = x + f(x) is neither pre-norm nor post-norm. It destroys the residual path while looking correct.
Comparing a pre-norm and post-norm model without matching the schedule. Post-norm with warm-up against pre-norm without it is not a comparison of normalisation placement.
Assuming a checkpoint's placement from the model family name. BERT is post-norm; GPT-2 is pre-norm; both are "transformers". Read the code.
Try it yourself
Change L from 24 to 8 and rerun. The gap between the two designs shrinks noticeably. Then set lr=3e-3 at L=24 and see which one degrades. Depth and learning rate are the two dials that make the difference visible, and both point the same way.
What to learn next
- RMSNorm — the normalisation that replaced LayerNorm in most current models.
- The residual stream — the path this decision either protects or interrupts.
- Gradient clipping — the other standard defence against a diverging run.
Researcher — Mathematics and papers.
The two formulations
$$ \text{Post-LN:} \quad x^{(\ell)} = \operatorname{LN}!\left( x^{(\ell-1)} + f_\ell(x^{(\ell-1)}) \right) $$
$$ \text{Pre-LN:} \quad x^{(\ell)} = x^{(\ell-1)} + f_\ell!\left( \operatorname{LN}(x^{(\ell-1)}) \right) $$
Only the pre-norm form admits the exact decomposition $x^{(L)} = x^{(0)} + \sum_\ell f_\ell(\cdot)$. Only it has $\partial x^{(\ell)} / \partial x^{(\ell-1)} = I + J_\ell$, with an exact identity term.
The gradient-scale result
Xiong et al. (2020), ICML 2020, arXiv:2002.04745, analyse both at initialisation. The analysis uses a mean-field approximation. Their central results:
- For post-LN, the expected gradient norm at the last layer's parameters is $O!\left( d \sqrt{\ln d} \right)$, independent of depth $L$. Gradients at early layers are attenuated.
- For pre-LN, the same quantity is $O!\left( d \sqrt{\ln d / L} \right)$, decreasing with depth and, crucially, comparable across layers.
One practical claim follows, and it changed common practice. Pre-LN transformers can be trained without a learning-rate warm-up. Xiong et al. demonstrate this on machine translation and BERT-style pretraining, reaching comparable quality with a shorter schedule.
Nguyen and Salazar (2019), arXiv:1910.05895, reached the same practical conclusion independently. Their setting was low-resource translation, alongside ScaleNorm and FixNorm.
Why post-norm can still win on quality
The pre-norm identity path is also a liability. Since the stream is never rescaled, its norm grows with depth while each sublayer's input is normalised. A sublayer's relative contribution therefore shrinks as $\ell$ increases. Late layers in a deep pre-norm model perturb the output less. The effect is sometimes described as behaving like a shallower ensemble of paths.
Wang et al. (2022), DeepNet, arXiv:2203.00555, formalise the trade. They derive an upper bound on the model update per step. A residual scaling $\alpha$ and initialisation $\beta$ are then chosen as functions of depth. Post-LN's update stays bounded independently of $L$. With $$ x^{(\ell)} = \operatorname{LN}!\left( \alpha\, x^{(\ell-1)} + f_\ell(x^{(\ell-1)}) \right) $$ they train a 1,000-layer encoder-decoder stably and report gains over pre-LN baselines at matched size.
Hybrid placements
- Sandwich norm, $x + \operatorname{LN}_2(f(\operatorname{LN}_1(x)))$, keeps the clean residual path and bounds the branch output. CogView (Ding et al., 2021, arXiv:2105.13290) uses it to control activation blow-up. So does the Gemma 2 report.
- NormFormer (Shleifer et al., 2021, arXiv:2110.09456) adds normalisation after attention and a learned per-head scale. It addresses a mismatch where pre-LN gradients at early layers exceed those at late layers.
- OLMo 2 and several other 2024-onward open models reorder the norms relative to the residual. It is reported as a stability measure, generally alongside QK normalisation. Isolating the contribution of any one of these changes from published reports is rarely possible.
The methodological caution
Normalisation placement co-varies with initialisation scale, learning rate, warm-up length, optimiser $\epsilon$, and precision. Narang et al. (2021), arXiv:2102.11972, found many published transformer modifications did not reproduce under a unified codebase. Pre-norm is unusual in that its benefit reproduces widely and has a clear mechanism. Claims about the finer variants deserve more scepticism.
Papers
- Vaswani et al., Attention Is All You Need, 2017 — arxiv.org/abs/1706.03762
- Nguyen and Salazar, Transformers without Tears, 2019 — arxiv.org/abs/1910.05895
- Xiong et al., On Layer Normalization in the Transformer Architecture, 2020 — arxiv.org/abs/2002.04745
- Shleifer et al., NormFormer, 2021 — arxiv.org/abs/2110.09456
- Wang et al., DeepNet: Scaling Transformers to 1,000 Layers, 2022 — arxiv.org/abs/2203.00555
What to learn next
- RMSNorm — the normalisation that replaced LayerNorm in most current models.
- The residual stream — the path this decision either protects or interrupts.
- Gradient clipping — the other standard defence against a diverging run.