Backpropagation

The chain rule, in the form actually used#

Let the network be a composition $f = f_L \circ f_{L-1} \circ \cdots \circ f_1$ with scalar loss $\mathcal{L}$. Write $h^{(l)}$ for the output of stage $l$, and define the adjoint:

$$ \bar{h}^{(l)} \;\equiv\; \frac{\partial \mathcal{L}}{\partial h^{(l)}} $$

The backward recursion is:

$$ \bar{h}^{(l-1)} = \left( \frac{\partial h^{(l)}}{\partial h^{(l-1)}} \right)^{!\top} \bar{h}^{(l)} $$

Where:

• $\dfrac{\partial h^{(l)}}{\partial h^{(l-1)}}$ — the Jacobian of stage $l$ with respect to its input.
• $\bar{h}^{(L)} = \dfrac{\partial \mathcal{L}}{\partial h^{(L)}}$ — the seed, from differentiating the loss.

For an affine layer $h^{(l)} = W^{(l)} h^{(l-1)} + b^{(l)}$ followed by elementwise $\phi$, this becomes:

$$ \bar{z}^{(l)} = \bar{h}^{(l)} \odot \phi'!\left(z^{(l)}\right), \qquad \nabla_{W^{(l)}} \mathcal{L} = \bar{z}^{(l)} \left(h^{(l-1)}\right)^{!\top}, \qquad \bar{h}^{(l-1)} = \left(W^{(l)}\right)^{!\top} \bar{z}^{(l)} $$

Where $\odot$ is the elementwise (Hadamard) product and $z^{(l)}$ is the pre-activation.

Why reverse mode, and not forward mode#

Both directions compute exact derivatives. The choice is purely about shape.

For $f : \mathbb{R}^{n} \to \mathbb{R}^{m}$:

• Forward mode propagates a Jacobian-vector product (JVP). One pass yields one column of the Jacobian. Full Jacobian costs $O(n)$ passes.
• Reverse mode propagates a vector-Jacobian product (VJP). One pass yields one row. Full Jacobian costs $O(m)$ passes.

Training has $n \sim 10^{9}$ parameters and $m = 1$ scalar loss. Reverse mode therefore costs a single pass; forward mode would cost a billion. This asymmetry, not any deeper property, is why every deep learning framework is built on reverse-mode automatic differentiation.

The cheap gradient principle — see Griewank and Walther (2008), Evaluating Derivatives — states that reverse-mode AD computes $\nabla f$ at a cost of at most $\approx 4 \times$ the cost of evaluating $f$, independent of $n$. In practice the backward pass runs at roughly $2 \times$ the forward FLOPs for dense layers.

The memory cost, which is the real constraint#

Reverse mode must retain intermediate activations from the forward pass to evaluate Jacobians on the way back. Peak activation memory is:

$$ O!\left(B \sum_{l=1}^{L} d_l\right) $$

Where $B$ is batch size and $d_l$ the width of layer $l$. This scales linearly in depth, and it — not parameter count — is what exhausts GPU memory when training large models.

Gradient checkpointing (Chen et al., 2016, Training deep nets with sublinear memory cost) trades compute for memory: store only $O(\sqrt{L})$ activations and recompute the rest during the backward pass, giving $O(\sqrt{L})$ memory at roughly $1.3 \times$ the compute.

Historical record#

The algorithm was derived independently several times before it reached deep learning:

• Linnainmaa (1970) — master's thesis giving reverse-mode accumulation of rounding error, the earliest complete statement.
• Werbos (1974) — PhD thesis applying it to neural networks.
• Rumelhart, Hinton and Williams (1986), Learning representations by back-propagating errors, Nature 323 — the paper that made the field adopt it.

Calling it "the 1986 algorithm" is inaccurate; 1986 is when it became known, not when it was invented.

Numerical gradient checking, done properly#

Use the central difference with relative error:

$$ \text{rel err} = \frac{\left| g_{\text{analytic}} - g_{\text{numeric}} \right|}{\max\left(\left|g_{\text{analytic}}\right|, \left|g_{\text{numeric}}\right|, \epsilon\right)} $$

Rules that matter:

• Use float64. In float32, catastrophic cancellation swamps the signal.
• Choose $h \approx 10^{-6}$. Total error is $O(h^2)$ from truncation plus $O(\epsilon_{\text{machine}}/h)$ from round-off, and $10^{-6}$ sits near the optimum for double precision.
• Below $10^{-7}$ relative error, the gradient is correct. Above $10^{-3}$, it is wrong.
• Check away from kinks. ReLU at exactly $z = 0$ is not differentiable and will produce a spurious failure.

torch.autograd.gradcheck implements this and should be your first move when writing a custom autograd.Function.

Known failure modes#

• Vanishing gradients — the product $\prod_l \phi'(z^{(l)}) |W^{(l)}|$ decays exponentially in $L$. Addressed by ReLU-family activations, residual connections (He et al., 2016), and normalisation layers.
• Exploding gradients — the same product diverging. Addressed by gradient clipping (Pascanu et al., 2013, On the difficulty of training recurrent neural networks).
• Weight transport — backpropagation requires the backward pass to use $\left(W^{(l)}\right)^{\top}$, the exact transpose of the forward weights. No biological mechanism is known for this. Lillicrap et al. (2016), Random synaptic feedback weights support error backpropagation, show that fixed random feedback matrices still permit learning, which weakens the biological objection without resolving it.

What to learn next#

• Gradient descent — what the network does with the gradients once it has them.
• Activation functions — the local slopes that scale the backward message.
• PyTorch basics — letting autograd do all of this for you.