Reward models

Bradley–Terry#

The standard reward model assumes preferences follow the Bradley–Terry model (Bradley and Terry, 1952). For prompt $x$ and completions $y_1, y_2$:

$$ p(y_1 \succ y_2 \mid x) = \frac{\exp r^(x,y_1)}{\exp r^(x,y_1) + \exp r^(x,y_2)} = \sigma!\left(r^(x,y_1) - r^*(x,y_2)\right) $$

$r^*$ is the latent reward and $\sigma$ the logistic function. Fitting $r_\phi$ by maximum likelihood over a preference dataset $\mathcal{D} = {(x, y_w, y_l)}$ gives

$$ \mathcal{L}R(\phi) = -\mathbb{E}{(x,y_w,y_l)\sim\mathcal{D}}\left[\log \sigma!\left(r_\phi(x,y_w) - r_\phi(x,y_l)\right)\right] $$

Two structural properties follow immediately and both matter downstream.

Shift invariance. $r_\phi$ and $r_\phi + c(x)$ give identical loss for any function of the prompt alone. The reward scale has no absolute meaning, and per-prompt offsets are unidentifiable. This is why PPO-style RLHF normalises advantages per prompt and why GRPO subtracts a group mean.

Transitivity is assumed, not observed. Bradley–Terry cannot represent intransitive human preferences ($A \succ B \succ C \succ A$), which real annotator populations exhibit. Munos et al., 2024 (Nash Learning from Human Feedback) and the SPPO line drop the BT assumption and solve for a Nash equilibrium of the preference game instead.

Where the signal ceiling comes from#

Inter-annotator agreement on open-ended preference data is typically 60–75%. InstructGPT (Ouyang et al., 2022) reported around 73% agreement between held-out human labellers, and their reward model's held-out accuracy was in the same range. A reward model cannot exceed the agreement rate of the labellers it was trained on, so accuracy well above that band is evidence of leakage or annotator-identity memorisation rather than quality.

RewardBench (Lambert et al., 2024) is the standard public benchmark. It reveals wide variance across categories — chat, reasoning, safety — and a persistent weakness on subtly-wrong reasoning, which is precisely the case where the reward signal is most needed.

Length bias, quantified#

The confounding demonstrated in the code is documented at scale.

• Singhal et al., 2023 (A Long Way to Go: Investigating Length Correlations in RLHF) found reward-model score correlates with response length at $r > 0.7$ on standard preference datasets, and that a large fraction of RLHF's apparent improvement is reproducible by optimising length alone.
• Dubois et al., 2024 (Length-Controlled AlpacaEval) built a regression-based debiasing of the evaluation itself, and showed length control substantially reorders the leaderboard.
• Park et al., 2024 (Disentangling Length from Quality in Direct Preference Optimization) show the same pathology arises in DPO without any explicit reward model.

Mitigations that are actually used: length-normalising the reward, adding a length penalty, balancing lengths within preference pairs at collection time, and reporting length-controlled win rates. None of them fully removes the effect.

Reward over-optimisation#

Gao et al., 2023 (Scaling Laws for Reward Model Overoptimization) is the key empirical result. Using a large "gold" reward model to label data for a smaller proxy reward model, they measured true reward as a function of KL distance $d = \sqrt{D_{\mathrm{KL}}(\pi | \pi_{\text{ref}})}$ from the reference policy, finding:

$$ R(d) = d\,(\alpha - \beta d) $$

for best-of-$n$ sampling, and a logarithmic analogue for RL. True reward rises, peaks, then falls as the policy exploits the proxy. The peak location scales with reward-model size and data; the shape does not go away. This is Goodhart's law with a fitted curve, and it is why the KL penalty exists.

Variants beyond a single scalar#

Lightman et al., 2023 (Let's Verify Step by Step) is the reference for process supervision: a PRM trained on 800k step-level labels solved 78% of a MATH test subset by reranking, substantially beating an outcome-supervised model. The cost is the label collection.

Rule-based verifiers are the most consequential recent shift. Where the answer can be checked mechanically — a unit test, a numeric match — the reward is exact, unhackable in the Bradley–Terry sense, and free. That is the foundation of the 2024–2025 reasoning models.

Papers#

• Bradley and Terry, Rank Analysis of Incomplete Block Designs, Biometrika 1952.
• Christiano et al., Deep Reinforcement Learning from Human Preferences, NeurIPS 2017 — arxiv.org/abs/1706.03741
• Stiennon et al., Learning to summarize from human feedback, NeurIPS 2020 — arxiv.org/abs/2009.01325
• Ouyang et al., Training language models to follow instructions with human feedback, 2022 — arxiv.org/abs/2203.02155
• Gao et al., Scaling Laws for Reward Model Overoptimization, ICML 2023 — arxiv.org/abs/2210.10760
• Lightman et al., Let's Verify Step by Step, 2023 — arxiv.org/abs/2305.20050
• Singhal et al., A Long Way to Go: Investigating Length Correlations in RLHF, 2023 — arxiv.org/abs/2310.03716
• Lambert et al., RewardBench, 2024 — arxiv.org/abs/2403.13787
• Munos et al., Nash Learning from Human Feedback, ICML 2024 — arxiv.org/abs/2312.00886

What to learn next#

• The KL penalty and the reference model — the leash that stops reward over-optimisation.
• Building preference data — where those comparisons come from.
• RLHF — the full loop this model sits inside.