RLHF — learning from human feedback

The pipeline, formally#

Stage 1 — supervised fine-tuning. Maximise $\mathbb{E}{(x,y)\sim\mathcal{D}{\text{SFT}}}[\log \pi(y \mid x)]$ on curated demonstrations, giving $\pi^{\text{SFT}}$.

Stage 2 — reward modelling. Under the Bradley-Terry model (1952), for a prompt $x$ and completions $y_w \succ y_l$: $$ p(y_w \succ y_l \mid x) = \frac{\exp r^(x, y_w)}{\exp r^(x,y_w) + \exp r^(x,y_l)} = \sigma!\left(r^(x,y_w) - r^(x,y_l)\right) $$ Fit $r_\psi$ by maximum likelihood: $$ \mathcal{L}R(\psi) = -\mathbb{E}{(x, y_w, y_l) \sim \mathcal{D}}\left[ \log \sigma!\left( r_\psi(x, y_w) - r_\psi(x, y_l) \right) \right] $$ $r^$ is identifiable only up to an additive function of $x$, so scores carry no absolute meaning.

Stage 3 — RL against the reward model. $$ \max_{\pi_\theta} \; \mathbb{E}{x \sim \mathcal{D},\, y \sim \pi\theta(\cdot \mid x)} \left[ r_\psi(x,y) \right] - \beta\, \mathbb{D}{\text{KL}}!\left[ \pi\theta(y\mid x) \,|\, \pi^{\text{SFT}}(y \mid x) \right] $$ Solved with PPO, treating each token as an action, the prefix as the state, and the sequence-level reward assigned at the final token with the per-token KL folded into the reward.

The KL term is not a regulariser#

It is part of the objective. The closed-form maximiser is $$ \pi^*(y \mid x) = \frac{1}{Z(x)} \pi^{\text{SFT}}(y \mid x) \exp!\left( \tfrac{1}{\beta} r_\psi(x,y) \right) $$ an exponential tilting of the reference policy. This identity is the foundation of DPO. It also shows $\beta$ selects a point on a frontier: $\beta \to \infty$ recovers $\pi^{\text{SFT}}$, and $\beta \to 0$ collapses onto the reward model's argmax, which is where reward hacking lives.

Reward over-optimisation#

Gao, Schulman and Hilton (2023) measured proxy reward against gold reward using a large reward model as ground truth. With $d = \sqrt{\mathrm{KL}(\pi | \pi^{\text{SFT}})}$, gold reward follows $$ R_{\text{gold}}(d) = d\,(\alpha - \beta d) $$ for best-of-$n$ sampling, and a logarithmic variant for RL. It rises, peaks, and then falls. Larger reward models peak later and higher, but every one of them turns over.

The practical consequence is that KL distance is a usable early-warning signal, and that there is an optimal amount of optimisation which more compute does not remove.

DPO and the offline family#

Rafailov et al. (2023) reparameterise the reward as $r(x,y) = \beta \log \frac{\pi_\theta(y \mid x)}{\pi^{\text{SFT}}(y\mid x)} + \beta \log Z(x)$. Substituting into the Bradley-Terry likelihood cancels $Z(x)$: $$ \mathcal{L}{\text{DPO}} = -\mathbb{E}\left[ \log \sigma!\left( \beta \log \frac{\pi\theta(y_w\mid x)}{\pi^{\text{SFT}}(y_w \mid x)} - \beta \log \frac{\pi_\theta(y_l \mid x)}{\pi^{\text{SFT}}(y_l\mid x)} \right) \right] $$

No reward model, no sampling, no RL. Known limitations: it is strictly offline, so it cannot correct itself on states it visits after updating; it is prone to reducing the likelihood of both completions in a pair; and the implicit reward is only well-behaved on the support of the preference data. Variants address these — IPO (Azar et al., 2023) removes the Bradley-Terry assumption, KTO (Ethayarajh et al., 2024) needs only binary good/bad labels, and ORPO folds the SFT stage in.

Removing the value network#

For reasoning tasks with a verifiable answer, a learned critic over token sequences is expensive and badly conditioned. GRPO (Shao et al., 2024) samples a group of $G$ completions per prompt and uses the group's own statistics as the baseline: $$ \hat{A}_i = \frac{r_i - \operatorname{mean}(r_1 \dots r_G)}{\operatorname{std}(r_1 \dots r_G)} $$ This is REINFORCE with a group baseline plus PPO clipping. Combined with rule-based rewards — does the code compile, is the final answer correct — it removed the reward model from the loop entirely for a large class of tasks, and it underpins much of the recent work on reasoning models.

Scaling the feedback#

Human labelling is the bottleneck. Two directions:

• Constitutional AI / RLAIF (Bai et al., 2022; Lee et al., 2023) — an LLM produces the preference labels against a written set of principles. Matches human-labelled RLHF on several benchmarks. It relocates the value judgements into the written constitution, which is arguably an improvement in transparency.
• Process supervision (Lightman et al., 2023) — label each reasoning step rather than the final answer. Substantially outperforms outcome supervision on mathematical reasoning, at much higher labelling cost.

Known failure modes, with citations#

• Length bias. Singhal et al. (2023) show reward models correlate strongly with length and that much of the apparent gain from RLHF is explained by it.
• Sycophancy. Sharma et al. (2023) demonstrate that preference training increases agreement with a user's stated view, including when it is wrong, because annotators prefer agreement.
• Annotator disagreement. InstructGPT reports inter-annotator agreement near 73%, which is a ceiling on reward-model accuracy.
• Distributional narrowing. RLHF measurably reduces output diversity. For creative tasks this is a loss, and it is why some products expose a less-tuned mode.
• Open problems. Casper et al. (2023), Open Problems and Fundamental Limitations of RLHF, is the systematic catalogue and the right starting point for anyone treating RLHF as solved.

References#

• Christiano et al. (2017), Deep Reinforcement Learning from Human Preferences — arxiv.org/abs/1706.03741.
• Stiennon et al. (2020), Learning to summarize from human feedback — arxiv.org/abs/2009.01325.
• Ouyang et al. (2022), Training language models to follow instructions with human feedback — arxiv.org/abs/2203.02155.
• Bai et al. (2022), Constitutional AI: Harmlessness from AI Feedback — arxiv.org/abs/2212.08073.
• Gao, Schulman and Hilton (2023), Scaling Laws for Reward Model Overoptimization — arxiv.org/abs/2210.10760.
• Rafailov et al. (2023), Direct Preference Optimization — arxiv.org/abs/2305.18290.
• Casper et al. (2023), Open Problems and Fundamental Limitations of RLHF — arxiv.org/abs/2307.15217.
• Shao et al. (2024), DeepSeekMath (GRPO) — arxiv.org/abs/2402.03300.

What to learn next#

• Fine-tuning — stage one of the pipeline, in detail.
• PPO — the algorithm doing stage three.
• Hallucination — the failure RLHF reduces but does not remove.