Catastrophic forgetting

The mechanism#

Catastrophic interference was identified in connectionist networks by McCloskey and Cohen, 1989 and Ratcliff, 1990. The cause is representational overlap: gradient descent on task B moves shared parameters without any term in the loss preserving task A's function.

Formally, after training on A the parameters sit at $\theta_A^$, a minimiser of $\mathcal{L}_A$. Training on B follows $-\nabla\mathcal{L}_B$, which has no component constraining $\mathcal{L}_A$. Since $\theta_A^$ lies on a low-dimensional manifold of solutions for A, almost any direction leaves it.

The corollary that matters practically: there generally exists a $\theta$ good at both — the 50%-replay row demonstrates one — but sequential gradient descent will not find it, because nothing in the sequential objective points there.

What actually preserves knowledge#

Replay / rehearsal. Mix samples from the old distribution into the new batches. This is not a heuristic: it makes the objective $\mathcal{L}_A + \mathcal{L}_B$ again, which is the joint problem whose solution you want. Everything else is an approximation of it.

Elastic Weight Consolidation (Kirkpatrick et al., 2017) approximates the old task's loss with a quadratic penalty weighted by the diagonal Fisher information:

$$ \mathcal{L}(\theta) = \mathcal{L}_B(\theta) + \frac{\lambda}{2}\sum_i F_i\,(\theta_i - \theta^*_{A,i})^2 $$

$F_i$ estimates how much task A's log-likelihood depends on parameter $i$, so parameters that mattered to A are held rigid and the rest are free. It works, and it needs a Fisher estimate over the old data — which means you still need the old data, or a saved Fisher diagonal.

Related: Synaptic Intelligence (Zenke et al., 2017) accumulates importance online; Learning without Forgetting (Li and Hoiem, 2017) distils from the pre-fine-tuning model's outputs on the new data, which uniquely requires no old data at all.

Measured on real LLMs#

• Luo et al., 2024 (An Empirical Study of Catastrophic Forgetting in LLMs During Continual Fine-tuning) find forgetting of domain knowledge, reasoning and reading comprehension worsens monotonically with model size from 1B to 7B, and that decoder-only models forget more than encoder–decoder ones.
• Qi et al., 2024 (Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!, ICLR 2024) is the alarming result: safety alignment of GPT-3.5 Turbo was removed by fine-tuning on 10 adversarially designed examples for under $0.20 through the public API; and fine-tuning on entirely benign datasets also measurably degraded safety.
• Biderman et al., 2024 (LoRA Learns Less and Forgets Less) quantify the adapter trade-off: LoRA better preserves out-of-domain performance and learns less in-domain, with the effect explained by the low rank of its updates.
• Ibrahim et al., 2024 (Simple and Scalable Strategies to Continually Pre-train LLMs) show that learning-rate re-warming plus re-decaying, combined with replaying a small fraction (as little as 5%) of the previous distribution, matches full retraining on the union of datasets at a fraction of the cost.

That last result is the most actionable in this list: LR re-warming plus ~5% replay is the current default recipe for continued pretraining.

Task vectors and merging as a mitigation#

Defining the task vector $\tau = \theta_{\text{ft}} - \theta_{\text{base}}$ (Ilharco et al., 2023), a scaled interpolation

$$ \theta = \theta_{\text{base}} + \lambda\tau, \qquad \lambda \in (0, 1) $$

trades new-task performance against retained general ability, and $\lambda \approx 0.5$–$0.8$ frequently keeps most of the fine-tuning gain while recovering much of what was lost. Wortsman et al., 2022 (Robust fine-tuning of zero-shot models, WiSE-FT) established this for CLIP, reporting improved robustness and accuracy from the interpolation. See merging model weights.

The theoretical floor#

Knoblauch et al., 2020 (Optimal Continual Learning has Perfect Memory and is NP-hard) prove that optimal continual learning requires perfect memory of all previous tasks, and that the resulting set-intersection problem is NP-hard. This is why every practical method is a trade-off rather than a fix, and why replay — which is literally partial memory — remains the strongest approach.

Papers#

• McCloskey and Cohen, Catastrophic Interference in Connectionist Networks, 1989 — Psychology of Learning and Motivation 24.
• French, Catastrophic forgetting in connectionist networks, 1999 — Trends in Cognitive Sciences 3(4).
• Kirkpatrick et al., Overcoming catastrophic forgetting in neural networks (EWC), PNAS 2017 — arxiv.org/abs/1612.00796
• Li and Hoiem, Learning without Forgetting, TPAMI 2017 — arxiv.org/abs/1606.09282
• Knoblauch et al., Optimal Continual Learning has Perfect Memory and is NP-hard, ICML 2020 — arxiv.org/abs/2006.05188
• Wortsman et al., Robust fine-tuning of zero-shot models (WiSE-FT), CVPR 2022 — arxiv.org/abs/2109.01903
• Ilharco et al., Editing Models with Task Arithmetic, ICLR 2023 — arxiv.org/abs/2212.04089
• Qi et al., Fine-tuning Aligned Language Models Compromises Safety, ICLR 2024 — arxiv.org/abs/2310.03693
• Luo et al., An Empirical Study of Catastrophic Forgetting in LLMs During Continual Fine-tuning, 2024 — arxiv.org/abs/2308.08747
• Ibrahim et al., Simple and Scalable Strategies to Continually Pre-train LLMs, TMLR 2024 — arxiv.org/abs/2403.08763
• Biderman et al., LoRA Learns Less and Forgets Less, TMLR 2024 — arxiv.org/abs/2405.09673

What to learn next#

• Merging model weights — recovering lost abilities by averaging.
• Adapters beyond LoRA — the structural way to limit the damage.
• Ablation studies — how to find out what your change actually cost.