Section 20

Direct Preference Optimization

Collapsing RLHF into a single loss

Paper: Direct Preference Optimization: Your Language Model is Secretly a Reward Model — Rafailov et al., 2023

By 2023, the RLHF recipe was influential but operationally involved: train a reward model, then run a PPO loop with a policy, reference, reward model, and often a critic. Those are logical roles rather than invariably four full independent networks, but the online sampling loop adds real complexity. Then a paper arrived with a striking proposal: optimize a supervised-style loss on preference pairs without an explicit reward model or on-policy RL loop. This is Direct Preference Optimization.

The trick in one sentence

DPO’s insight is that the reward model and the policy are not really two separate things. The RLHF objective already implies a relationship between them, and once you write that relationship down, you can express the reward entirely in terms of the policy, substitute it into the preference loss, and the reward model simply vanishes. The policy, it turns out, is secretly a reward model. Let’s derive it, because the derivation is short and the payoff is the whole chapter.

Step 1: the optimal RLHF policy has a closed form

Recall the RLHF objective from the PPO chapter: maximize reward while staying close to the reference, with a KL penalty of strength $\beta$ holding you near $\pi_{\text{ref}}$ :

\max_{\pi}\ \mathbb{E}_{x,\,y\sim\pi}\big[\,r(x,y)\,\big] \;-\; \beta\,\mathrm{KL}\!\big(\pi(\cdot|x)\,\big\|\,\pi_{\text{ref}}(\cdot|x)\big).

This particular objective has a known closed-form solution. The policy that maximizes it is the reference distribution, reweighted by the exponentiated reward:

\pi^{*}(y|x) \;=\; \frac{1}{Z(x)}\,\pi_{\text{ref}}(y|x)\,\exp\!\Big(\tfrac{1}{\beta}\,r(x,y)\Big),

where $Z(x) = \sum_{y}\pi_{\text{ref}}(y|x)\exp(r(x,y)/\beta)$ is a normalizing constant (the partition function) that makes it a valid distribution. Intuitively: start from the reference, then up-weight high-reward answers and down-weight low-reward ones, with $\beta$ controlling how aggressively. This is just the familiar fact that the KL-regularized reward objective is solved by a softmax-like Boltzmann distribution.

Step 2: invert it to read off the reward

Here’s the move. That equation relates the optimal policy to the reward. So solve it for the reward. Take logs and rearrange:

r(x,y) \;=\; \beta\,\log\frac{\pi^{*}(y|x)}{\pi_{\text{ref}}(y|x)} \;+\; \beta\,\log Z(x).

Read that carefully: it is the heart of DPO. The reward of any answer is just $\beta$ times the log-ratio between the optimal policy and the reference, plus a term $\beta\log Z(x)$ that depends only on the prompt $x$ , not on the answer $y$ . The policy encodes the reward. If you know the optimal policy and the reference, you already know the reward up to a per-prompt constant. We call this quantity the implicit reward: $\hat r(x,y) = \beta\log\frac{\pi_\theta(y|x)}{\pi_{\text{ref}}(y|x)}$ .

Step 3: substitute into the preference likelihood

Now bring in the Bradley–Terry model from the reward-models chapter. It says the probability that a “winning” answer $y_w$ beats a “losing” one $y_l$ is a logistic function of their reward difference:

P(y_w \succ y_l \mid x) \;=\; \sigma\!\big(r(x,y_w) - r(x,y_l)\big).

Substitute our expression for $r$ from Step 2. The magic: the troublesome $\beta\log Z(x)$ term depends only on $x$ , so it is identical for $y_w$ and $y_l$ and cancels in the difference. The intractable partition function (the thing that normally forces you into RL) disappears completely. What’s left is written purely in terms of the policy and the reference:

P(y_w \succ y_l \mid x) \;=\; \sigma\!\Big(\beta\log\frac{\pi_\theta(y_w|x)}{\pi_{\text{ref}}(y_w|x)} - \beta\log\frac{\pi_\theta(y_l|x)}{\pi_{\text{ref}}(y_l|x)}\Big).

Maximize the likelihood of the observed preferences (equivalently, minimize its negative log) and you have the DPO loss:

\mathcal{L}_{\text{DPO}} \;=\; -\,\mathbb{E}_{(x,y_w,y_l)}\Big[\log\sigma\Big(\beta\log\tfrac{\pi_\theta(y_w|x)}{\pi_{\text{ref}}(y_w|x)} - \beta\log\tfrac{\pi_\theta(y_l|x)}{\pi_{\text{ref}}(y_l|x)}\Big)\Big].

That’s it. A plain supervised loss over preference triples $(x, y_w, y_l)$ : no reward model, no sampling, no RL loop, no critic. You compute four log-probabilities (the chosen and rejected answers under both $\pi_\theta$ and the frozen $\pi_{\text{ref}}$ ), form the difference, push it through $-\log\sigma$ , and backpropagate.

What the gradient actually does

The DPO gradient has a clean interpretation. It increases the likelihood of the chosen answer $y_w$ and decreases that of the rejected answer $y_l$ . But each pair is weighted by $\sigma\big(\hat r(x,y_l) - \hat r(x,y_w)\big)$ , how wrong the implicit reward currently is. Pairs the model already ranks correctly contribute almost nothing; pairs it gets backwards get the biggest push. So DPO automatically spends its gradient where the implicit reward most disagrees with the human label: a built-in hard-example weighting, with no reward model in sight.

Try it

Toggle between the two-stage RLHF pipeline (reward model + PPO loop) and the single DPO loss on the same preference pair. Watch the implicit reward $\beta\log\frac{\pi_\theta}{\pi_{\text{ref}}}$ move the chosen and rejected log-probabilities apart, and see that it lands in the same place the RM-plus-PPO route was trying to reach, just without ever building the reward model.

RLHF vs DPO: same goal, fewer stages

One preference pair — a chosen response y_w and a rejected one y_l — optimized two different ways.

RLHF

3 stages

Collect preferences

(y_w ≻ y_l) pairs

Train reward model

fit r̂(x, y) via Bradley–Terry

PPO RL loop

sample → score with r̂ → KL-reg update → repeat

DPO

1 stage

One supervised loss on the pair

−log σ( β·log π(y_w)/π_ref(y_w) − β·log π(y_l)/π_ref(y_l) )

no reward model · no RL loop

DPO implicit reward · r(x, y) = β · log( π(y|x) / π_ref(y|x) )

β = 0.10 (strength of the KL constraint to π_ref)

y_w (chosen)r = 0.140 (log-ratio 1.40)

y_l (rejected)r = -0.080 (log-ratio -0.80)

implicit reward margin r(y_w) − r(y_l)+0.220

DPO optimizes the same objective as RLHF but skips the explicit reward model and the RL loop by using a closed-form loss. The reward is implicit in the policy-versus-reference log-ratio: r(x, y) = β·log(π/π_ref). Both pipelines are ultimately doing the same thing — increasing the margin between the chosen and rejected responses — but DPO collapses the three RLHF stages into one supervised step.

What DPO trades away

DPO is often simpler and cheaper to implement than PPO-based RLHF, which has made it widely used in open post-training. Its empirical stability and quality still depend on the data, model, hyperparameters, and evaluation. The simplification is not free, and the trade is worth naming precisely.

DPO is usually run offline, learning from a fixed set of preference pairs without generating and scoring new samples during that run. PPO, by contrast, is on-policy: it samples the current policy and receives a reward on those samples. A fixed offline DPO run has no direct feedback on candidates outside its dataset, though the learned policy can still generalize and teams can iterate DPO with newly generated and labeled data.

This can make DPO sensitive to distribution shift. As training moves $\pi_\theta$ away from the policy that generated preference pairs, a fixed dataset may provide less representative supervision. PPO can re-sample and re-score, while an offline DPO run is limited to its snapshot. This is one motivation for careful data mixtures, regularization, variants, and iterative collection of fresh pairs.

The next chapter tours the DPO zoo — IPO, KTO, ORPO, SimPO — each one targeting a specific weakness we just identified: the deterministic-preference overfitting, the need for paired data, the separate SFT stage, and the lingering length bias from the previous chapter.