The feedforward layer

Definition#

$$ \operatorname{FFN}(x) = W_2 \, \phi(W_1 x + b_1) + b_2 $$

with $W_1 \in \mathbb{R}^{d_{\text{ff}} \times d}$, $W_2 \in \mathbb{R}^{d \times d_{\text{ff}}}$ and $\phi$ an elementwise nonlinearity. Applied identically at every position, so it is a position-wise map $\mathbb{R}^d \to \mathbb{R}^d$ lifted over the sequence axis. Equivalently a $1 \times 1$ convolution over the token axis, which is how the original paper describes it.

Parameters: $2 d\, d_{\text{ff}} + d_{\text{ff}} + d$. At $d_{\text{ff}} = 4d$ that is $8d^2$, against $4d^2$ for attention, giving the two-thirds share.

FLOPs per token: $4 d\, d_{\text{ff}} = 16 d^2$, independent of sequence length. Attention's cost is $8d^2 + 4Td$, so the feedforward layer dominates until $T$ becomes comparable to $6d$.

The key-value memory reading#

Geva et al. (2021), Transformer Feed-Forward Layers Are Key-Value Memories, arXiv:2012.14913, decompose the layer as

$$ \operatorname{FFN}(x) = \sum_{i=1}^{d_{\text{ff}}} \phi(k_i \cdot x + b_i) \, v_i $$

where $k_i$ is the $i$-th row of $W_1$ and $v_i$ the $i$-th column of $W_2$. The output is a coefficient-weighted sum of the $v_i$. Coefficients are set by how well $x$ matches each $k_i$.

Empirically, individual $k_i$ correspond to human-interpretable input patterns. Early layers hold shallow n-gram patterns, later layers semantic topics. The corresponding $v_i$, projected through the output head, induce interpretable shifts in the next-token distribution. Geva et al. (2022), arXiv:2203.14680, extend this to show the update is a sub-update in vocabulary space.

Follow-on work built on this reading:

• Meng et al. (2022), ROME, arXiv:2202.05262, localise factual associations to mid-layer feedforward modules via causal tracing. They edit them with a rank-one update to $W_2$.
• Dai et al. (2022), Knowledge Neurons, arXiv:2104.08696, attribute specific facts to individual hidden units.

The caution is the same one that applies throughout mechanistic interpretability. Locating where an intervention succeeds is not the same as locating where a fact is stored. Hase et al. (2023), arXiv:2301.04213, show editing succeeds at layers that causal tracing does not identify. That weakens the localisation claim considerably.

Sparsity#

In trained ReLU transformers, the fraction of hidden units active per token is typically a few percent. Li et al. (2023), The Lazy Neuron Phenomenon, arXiv:2210.06313, document this across scales. It strengthens with size. Three practical consequences:

• Conditional computation is viable: skipping inactive units saves real work (Liu et al., 2023, Deja Vu, arXiv:2310.17157).
• Mixture-of-experts is a structured form of the same idea. Each token is routed to a few of many feedforward experts. See Shazeer et al. (2017) and Fedus et al. (2021).
• GELU and SiLU produce small nonzero values rather than exact zeros. The sparsity is approximate, and thresholding is required to exploit it.

Depth against width in the feedforward layer#

The layer is a one-hidden-layer MLP, and one hidden layer is the minimum for universal approximation. Making it deeper is possible, and essentially nobody does it. The residual stack already provides depth. A wide shallow MLP maps far better onto matrix-multiply hardware.

Raising $d_{\text{ff}} / d$ above 4 is more common than adding depth. Kaplan et al. (2020), arXiv:2001.08361, report a robust finding. Performance depends far more on total non-embedding parameter count than on aspect ratios. That holds within a wide band. It reasonably justifies a ratio that has stayed at 4 for eight years without much scrutiny.

Papers#

• Vaswani et al., Attention Is All You Need, 2017 — arxiv.org/abs/1706.03762
• Geva et al., Transformer Feed-Forward Layers Are Key-Value Memories, 2021 — arxiv.org/abs/2012.14913
• Meng et al., Locating and Editing Factual Associations in GPT, 2022 — arxiv.org/abs/2202.05262
• Hase et al., Does Localization Inform Editing?, 2023 — arxiv.org/abs/2301.04213
• Li et al., The Lazy Neuron Phenomenon, 2023 — arxiv.org/abs/2210.06313

What to learn next#

• SwiGLU and gated feedforward layers — the version nearly every current model uses.
• Activation functions — the squashing step, in its own right.
• Counting a model's parameters by hand — putting the two-thirds share into real numbers.