Direct preference optimisation

The derivation, in three steps#

Step 1. The RLHF objective has the closed-form solution derived in the KL penalty lesson:

$$ \pi^*(y\mid x) = \frac{1}{Z(x)}\pi_{\text{ref}}(y\mid x)\exp!\left(\tfrac{1}{\beta}r(x,y)\right) $$

Step 2. Rearrange for the reward:

$$ r(x,y) = \beta \log \frac{\pi^*(y \mid x)}{\pi_{\text{ref}}(y\mid x)} + \beta \log Z(x) $$

Step 3. Substitute into the Bradley–Terry likelihood. The preference probability depends only on the difference of two rewards at the same prompt, and $\beta \log Z(x)$ is identical for both — so it cancels:

$$ p(y_w \succ y_l \mid x) = \sigma!\left(\beta\log\frac{\pi^(y_w|x)}{\pi_{\text{ref}}(y_w|x)} - \beta\log\frac{\pi^(y_l|x)}{\pi_{\text{ref}}(y_l|x)}\right) $$

Maximum likelihood on this gives the DPO loss:

$$ \mathcal{L}{\text{DPO}} = -\mathbb{E}{(x,y_w,y_l)}\left[\log\sigma!\left(\beta\log\frac{\pi_\theta(y_w|x)}{\pi_{\text{ref}}(y_w|x)} - \beta\log\frac{\pi_\theta(y_l|x)}{\pi_{\text{ref}}(y_l|x)}\right)\right] $$

The intractable partition function $Z(x)$ — the reason RLHF needed RL — disappears because it is prompt-dependent and reward-difference-invariant. That single cancellation is the entire contribution of the paper (Rafailov et al., 2023).

The gradient, and what it says about the toy result#

$$ \nabla_\theta \mathcal{L}{\text{DPO}} = -\beta\,\mathbb{E}\Big[\underbrace{\sigma!\left(\hat r\theta(x,y_l) - \hat r_\theta(x,y_w)\right)}{\text{weight: high when the model is wrong}}\big(\nabla\theta\log\pi_\theta(y_w|x) - \nabla_\theta\log\pi_\theta(y_l|x)\big)\Big] $$

with $\hat r_\theta = \beta\log\frac{\pi_\theta}{\pi_{\text{ref}}}$. Two consequences.

Automatic example weighting. Pairs the model already ranks correctly contribute almost nothing. This is what makes DPO stable without a value network.

Only the difference of gradients is controlled. Nothing in the expression forces $\log \pi_\theta(y_w)$ upward in absolute terms. The toy run's fall from −1.77 to −2.73 is the objective behaving exactly as written.

Likelihood displacement#

That last effect has a name and a literature.

Pal et al., 2024 (Smaug: Fixing Failure Modes of Preference Optimisation with DPO-Positive) show that when $y_w$ and $y_l$ have small edit distance, DPO's gradient reduces $\log\pi_\theta(y_w)$, and propose DPOP — adding a term penalising the chosen log-probability dropping below the reference.

Razin et al., 2024 (Unintentional Unalignment: Likelihood Displacement in Direct Preference Optimization) go further and characterise where the displaced mass goes. It flows to responses with similar embeddings to the rejected one, which can include responses semantically opposite to the chosen one. Their headline case: training a model to refuse unsafe requests via DPO increased its unsafe-compliance rate from 4.6% to 94.4%. They introduce a centered hidden embedding similarity (CHES) score to identify and filter the pairs responsible.

This is the strongest practical reason to log logps/chosen throughout a DPO run.

The variant zoo, and what each fixes#

TRL exposes most of these as loss_type on a single DPOConfig, and supports weighted combinations via loss_type=[...] with loss_weights=[...].

Does DPO match PPO?#

The honest answer is: usually, and not always, and the gap is about data rather than about the loss.

• Ivison et al., 2024 (Unpacking DPO and PPO) ran a controlled comparison and found PPO ahead on reasoning-heavy tasks, attributing most of the gap to PPO's online sampling.
• Xu et al., 2024 (Is DPO Superior to PPO for LLM Alignment?) reach the same conclusion more sharply, arguing DPO's offline nature leaves it exposed to distribution shift, and that PPO's advantage is largest exactly where the SFT policy is weakest.
• Tajwar et al., 2024 (Preference Fine-Tuning of LLMs Should Leverage Suboptimal, On-Policy Data) isolate the mechanism: on-policy sampling and negative gradients are what matter, not which loss you write down.

The practical synthesis, and what most open recipes now do: iterative DPO. Sample from the current policy, label the pairs (by humans, a reward model, or an LLM judge), run DPO, repeat. This recovers most of PPO's benefit while keeping DPO's simplicity, and it is what Llama 3's post-training and Tülu 3 both use.

Papers#

• Rafailov et al., Direct Preference Optimization: Your Language Model is Secretly a Reward Model, NeurIPS 2023 — arxiv.org/abs/2305.18290
• Azar et al., A General Theoretical Paradigm to Understand Learning from Human Preferences (IPO), AISTATS 2024 — arxiv.org/abs/2310.12036
• Ethayarajh et al., KTO: Model Alignment as Prospect Theoretic Optimization, ICML 2024 — arxiv.org/abs/2402.01306
• Hong et al., ORPO: Monolithic Preference Optimization without Reference Model, EMNLP 2024 — arxiv.org/abs/2403.07691
• Pal et al., Smaug: Fixing Failure Modes of Preference Optimisation with DPO-Positive, 2024 — arxiv.org/abs/2402.13228
• Meng et al., SimPO, NeurIPS 2024 — arxiv.org/abs/2405.14734
• Tajwar et al., Preference Fine-Tuning of LLMs Should Leverage Suboptimal, On-Policy Data, ICML 2024 — arxiv.org/abs/2404.14367
• Ivison et al., Unpacking DPO and PPO, NeurIPS 2024 — arxiv.org/abs/2406.09279
• Xu et al., Is DPO Superior to PPO for LLM Alignment?, ICML 2024 — arxiv.org/abs/2404.10719
• Razin et al., Unintentional Unalignment: Likelihood Displacement in DPO, ICLR 2025 — arxiv.org/abs/2410.08847

What to learn next#

• Building preference data — the pairs this method eats.
• GRPO — the online method that replaced PPO for reasoning.
• Reward models — the component DPO removes, and why it still matters.