Inside a Transformer Section 076
Inside a Transformer Block
A transformer is one small block repeated many times. This section takes that block apart piece by piece.
12 of 12 lessons published Three reading levels on every lesson
Lessons in order
Work top to bottom. Each lesson assumes the one above it.
- Walking through one transformer block
- The residual stream
- Layer normalisation
- Pre-norm vs post-norm
- RMSNorm
- The feedforward layer
- SwiGLU and gated feedforward layers
- Tied input and output embeddings
- Decoder-only vs encoder-decoder
- Counting a model's parameters by hand
- FLOPs per token
- Building a tiny GPT from scratch