Fine-tuning

The objective#

Supervised fine-tuning is ordinary maximum likelihood on a conditional distribution, with the loss masked to the response:

L(θ) = − Σ_{i}  Σ_{t ∈ response_i}  log p_θ ( y_t | y_{<t}, prompt_i )

θ are the model parameters, prompt_i the input tokens of example i, y_t the t-th token of the target response. Tokens in the prompt receive no loss. The pretraining objective is identical apart from the mask covering everything, which is why SFT is best understood as a continuation of pretraining on a narrow, curated distribution rather than a different algorithm.

Memory arithmetic for full fine-tuning#

Per parameter, with Adam and mixed precision:

bf16 weights                    2 bytes
bf16 gradients                  2 bytes
fp32 master weights             4 bytes
fp32 Adam first moment          4 bytes
fp32 Adam second moment         4 bytes
                               ---------
                               16 bytes / parameter

A 7B model therefore needs about 112 GB before a single activation is stored, which exceeds any single accelerator commonly available. Activations add O(layers × batch × sequence × hidden) on top. This arithmetic, not accuracy, is the reason parameter-efficient methods dominate practice — see LoRA.

Catastrophic forgetting#

McCloskey and Cohen (1989) named the phenomenon in connectionist networks: sequential training on task B degrades task A, because the parameters encoding A are freely reused for B. It has never been solved, only managed.

Three families of mitigation, in increasing order of practical use:

• Regularisation towards the old parameters. Elastic Weight Consolidation (Kirkpatrick et al., 2017) adds Σ_i F_i (θ_i − θ*_i)², where θ* are the pretrained parameters and F_i the diagonal Fisher information estimating how much parameter i mattered to the old task. Principled, and rarely used at LLM scale because estimating F is expensive.
• Replay. Mixing a fraction of general-purpose data into the fine-tuning mixture. Crude, cheap, effective; 5 to 30 percent is the usual range.
• Constrained capacity. Restrict the update to a low-rank or otherwise small subspace. Biderman et al. (2024), LoRA Learns Less and Forgets Less, quantify the trade directly: LoRA underperforms full fine-tuning on target-domain gains, and preserves base-model capability substantially better.

Data quality dominates data quantity#

Zhou et al. (2023), LIMA: Less Is More for Alignment, fine-tuned LLaMA-65B on 1,000 carefully curated prompt–response pairs with no RLHF, and reached responses preferred to or equal with GPT-4's in 43% of a human comparison. Their superficial alignment hypothesis is the load-bearing claim: knowledge and capability are acquired almost entirely during pretraining, and alignment tuning mainly selects which subdistribution of formats and styles the model emits.

This reframes what SFT is for. If capability is already present, adding examples past the point where the format is unambiguous buys little and risks forgetting. It also explains the persistent failure of SFT as a knowledge-injection mechanism: you are selecting a style, not writing to a store.

Gekhman et al. (2024) sharpen this with a measured mechanism: fine-tuning examples containing facts absent from pretraining are learned slowly, and as the model fits them it becomes measurably more prone to hallucinating on other questions. Teaching new facts by SFT trains the behaviour of asserting unsupported claims.

Beyond supervised fine-tuning#

RLHF (Ouyang et al., 2022) trains a reward model on human pairwise preferences, then optimises the policy with PPO under a KL penalty against the SFT reference:

max_θ  E [ r_φ(x, y) ]  −  β · D_KL( π_θ(y|x) ‖ π_ref(y|x) )

r_φ is the learned reward, π_ref the frozen SFT model, β the strength of the anchor. The KL term is what stops reward hacking from destroying fluency.

DPO (Rafailov et al., 2023) shows this objective has a closed-form optimal policy, allowing the reward model to be eliminated and preferences to be optimised with a simple classification loss on preferred and rejected pairs. Far cheaper and far more stable; it is the default starting point for preference tuning now. Later variants (IPO, KTO, ORPO) adjust the loss shape or drop the need for paired data.

The practical pipeline is SFT for format and task, then preference optimisation for the qualities that are easier to compare than to demonstrate.

Hyperparameters that actually matter#

• Learning rate: 1e-5 to 2e-5 for full fine-tuning, 1e-4 to 3e-4 for LoRA adapters. The most common cause of a destroyed model is a learning rate borrowed from a pretraining recipe.
• Epochs: 1 to 3. Held-out loss usually turns upward before training loss looks suspicious.
• Warmup and cosine decay: standard, and worth keeping; a cold start at full learning rate on a converged model is needlessly destructive.
• Sequence packing: concatenating short examples to fill the context improves throughput, but requires correct attention masking or examples bleed into each other.

Evaluation#

Fine-tuning changes behaviour globally, so evaluate globally.

• A task set for the thing you trained for, scored automatically where possible.
• A regression set of general capability, to detect forgetting. Even a small held-out slice of general instruction data catches most damage.
• A safety set, because narrow fine-tuning is known to weaken alignment. Qi et al. (2023) demonstrated that fine-tuning on a small number of benign examples measurably degrades safety behaviour in aligned models.

Papers#

• Ouyang et al., Training Language Models to Follow Instructions with Human Feedback, 2022 — arxiv.org/abs/2203.02155
• Wei et al., Finetuned Language Models Are Zero-Shot Learners (FLAN), 2021 — arxiv.org/abs/2109.01652
• Zhou et al., LIMA: Less Is More for Alignment, 2023 — arxiv.org/abs/2305.11206
• Rafailov et al., Direct Preference Optimization, 2023 — arxiv.org/abs/2305.18290
• Kirkpatrick et al., Overcoming Catastrophic Forgetting in Neural Networks, 2017 — arxiv.org/abs/1612.00796
• Gekhman et al., Does Fine-Tuning LLMs on New Knowledge Encourage Hallucinations?, 2024 — arxiv.org/abs/2405.05904
• Qi et al., Fine-tuning Aligned Language Models Compromises Safety, 2023 — arxiv.org/abs/2310.03693
• Biderman et al., LoRA Learns Less and Forgets Less, 2024 — arxiv.org/abs/2405.09673

What to learn next#

• LoRA — fine-tuning by training a small add-on, which is how it is done in practice.
• What is RAG? — the right tool when the problem is missing facts, not missing behaviour.
• Hallucination — why teaching new facts by fine-tuning tends to make things worse.