RLHF vs DPO
A pretrained model predicts text. An assistant does what you asked. Alignment is the step in between, and for a few years it meant one pipeline: RLHF. DPO showed that most of that pipeline can be collapsed into a single supervised-style loss. Knowing both is how you decide when each is the right tool.
RLHF in three stages
- SFT. Fine-tune on demonstrations of good behaviour. This gives a reference policy $\pi_{ref}$.
- Reward model. Collect pairs where a human preferred response $y_w$ over $y_l$ for the same prompt $x$. Fit a scalar reward $r(x, y)$ with the Bradley-Terry loss:
$$\mathcal{L}_{RM} = -\log \sigma\big(r(x, y_w) - r(x, y_l)\big)$$
- RL. Optimise the policy against the reward, with a leash so it cannot wander far from $\pi_{ref}$:
$$\max_{\pi}\ \mathbb{E}{x,\ y \sim \pi}\big[r(x, y)\big] - \beta, \mathrm{KL}\big(\pi ,|, \pi{ref}\big)$$
PPO is the usual optimiser. In practice that means four large models in memory: the policy, the reference, the reward model and a value network (the critic).
The DPO trick
The KL-regularised objective has a closed-form optimum:
$$\pi^*(y \mid x) = \frac{1}{Z(x)}, \pi_{ref}(y \mid x), \exp!\big(r(x, y)/\beta\big)$$
Rearrange to express the reward in terms of the policy:
$$r(x, y) = \beta \log \frac{\pi^*(y \mid x)}{\pi_{ref}(y \mid x)} + \beta \log Z(x)$$
Put that into the Bradley-Terry loss. The term $\beta \log Z(x)$ depends only on the prompt, so it cancels in the difference. What is left needs no reward model and no sampling:
$$\mathcal{L}{DPO} = -\log \sigma!\left(\beta\left[\log\frac{\pi(y_w \mid x)}{\pi{ref}(y_w \mid x)} - \log\frac{\pi(y_l \mid x)}{\pi_{ref}(y_l \mid x)}\right]\right)$$
The model is the reward model: its implicit reward is $\beta \log(\pi/\pi_{ref})$.
A worked example
Sequence log-probabilities (sum over response tokens only), with $\beta = 0.1$:
| policy $\log \pi$ | reference $\log \pi_{ref}$ | ratio | |
|---|---|---|---|
| chosen $y_w$ | -12.0 | -13.0 | +1.0 |
| rejected $y_l$ | -15.0 | -14.0 | -1.0 |
Margin $= 0.1 \times (1.0 - (-1.0)) = 0.2$. Loss $= -\log \sigma(0.2) = 0.598$. At initialisation the policy equals the reference, the margin is 0 and the loss is $\ln 2 = 0.693$, so a healthy run starts at 0.693 and falls.
The gradient scales each pair by $\sigma(-\text{margin}) = 0.45$ here: pairs the implicit reward already ranks correctly get less weight, pairs it gets wrong get more. It pushes up $\log \pi(y_w)$ and down $\log \pi(y_l)$.
Code
import torch
import torch.nn.functional as F
def seq_logp(logits, labels, mask):
logp = torch.log_softmax(logits[:, :-1], dim=-1)
tok = logp.gather(-1, labels[:, 1:].unsqueeze(-1)).squeeze(-1)
return (tok * mask[:, 1:]).sum(-1)
def dpo_loss(pi_w, pi_l, ref_w, ref_l, beta=0.1):
margin = beta * ((pi_w - ref_w) - (pi_l - ref_l))
return -F.logsigmoid(margin).mean(), margin.detach()
loss, margin = dpo_loss(
torch.tensor([-12.0]), torch.tensor([-15.0]),
torch.tensor([-13.0]), torch.tensor([-14.0]),
)
print(loss.item(), margin.item())
This prints about 0.598 and 0.2. Precompute the reference log-probs once and cache them so you only keep one model on the GPU.
RLHF versus DPO
| RLHF with PPO | DPO | |
|---|---|---|
| Models in memory | policy, reference, reward, value | policy, reference (precomputable) |
| Data | online samples scored by a reward model | fixed preference pairs |
| Exploration | yes: finds behaviour not in the data | no: limited to the pairs |
| Stability | fiddly, many hyperparameters | simple, one knob ($\beta$) |
| Failure | reward hacking | likelihood displacement, length drift |
Use DPO for style, tone and safety preferences when you have good pairs. Use online RL when you have a reward you can compute and want the model to discover new behaviour, as in the reasoning-model recipes next week.
Known failure modes
- Reward hacking. The policy finds outputs the reward model likes and humans do not (longer, more confident, more flattering).
- Likelihood displacement. DPO can lower the probability of both responses, because the loss only constrains their difference.
- Verbosity drift. Preference data often favours longer answers. Check response length before and after.
Common mistakes
- Summing log-probs over the prompt as well as the response.
- Using a reference that is not the model you started from.
- Setting $\beta$ too low, so the policy drifts far from the reference.
- Judging success by the DPO loss. Look at held-out preference accuracy and read generations.
Check yourself
- Why does $Z(x)$ cancel?
- A pair has a scaled margin of $-0.4$ (already multiplied by $\beta$). What is the loss, and is its gradient weight larger or smaller than at margin $0.2$?
- What does raising $\beta$ do to how far the policy may move?
- Name one reason online RL can beat DPO on the same prompts.