RLHF vs DPO

A pretrained model predicts text. An assistant does what you asked. Alignment is the step in between, and for a few years it meant one pipeline: RLHF. DPO showed that most of that pipeline can be collapsed into a single supervised-style loss. Knowing both is how you decide when each is the right tool.

RLHF in three stages

  1. SFT. Fine-tune on demonstrations of good behaviour. This gives a reference policy $\pi_{ref}$.
  2. Reward model. Collect pairs where a human preferred response $y_w$ over $y_l$ for the same prompt $x$. Fit a scalar reward $r(x, y)$ with the Bradley-Terry loss:

$$\mathcal{L}_{RM} = -\log \sigma\big(r(x, y_w) - r(x, y_l)\big)$$

  1. RL. Optimise the policy against the reward, with a leash so it cannot wander far from $\pi_{ref}$:

$$\max_{\pi}\ \mathbb{E}{x,\ y \sim \pi}\big[r(x, y)\big] - \beta, \mathrm{KL}\big(\pi ,|, \pi{ref}\big)$$

PPO is the usual optimiser. In practice that means four large models in memory: the policy, the reference, the reward model and a value network (the critic).

The DPO trick

The KL-regularised objective has a closed-form optimum:

$$\pi^*(y \mid x) = \frac{1}{Z(x)}, \pi_{ref}(y \mid x), \exp!\big(r(x, y)/\beta\big)$$

Rearrange to express the reward in terms of the policy:

$$r(x, y) = \beta \log \frac{\pi^*(y \mid x)}{\pi_{ref}(y \mid x)} + \beta \log Z(x)$$

Put that into the Bradley-Terry loss. The term $\beta \log Z(x)$ depends only on the prompt, so it cancels in the difference. What is left needs no reward model and no sampling:

$$\mathcal{L}{DPO} = -\log \sigma!\left(\beta\left[\log\frac{\pi(y_w \mid x)}{\pi{ref}(y_w \mid x)} - \log\frac{\pi(y_l \mid x)}{\pi_{ref}(y_l \mid x)}\right]\right)$$

The model is the reward model: its implicit reward is $\beta \log(\pi/\pi_{ref})$.

A worked example

Sequence log-probabilities (sum over response tokens only), with $\beta = 0.1$:

policy $\log \pi$reference $\log \pi_{ref}$ratio
chosen $y_w$-12.0-13.0+1.0
rejected $y_l$-15.0-14.0-1.0

Margin $= 0.1 \times (1.0 - (-1.0)) = 0.2$. Loss $= -\log \sigma(0.2) = 0.598$. At initialisation the policy equals the reference, the margin is 0 and the loss is $\ln 2 = 0.693$, so a healthy run starts at 0.693 and falls.

The gradient scales each pair by $\sigma(-\text{margin}) = 0.45$ here: pairs the implicit reward already ranks correctly get less weight, pairs it gets wrong get more. It pushes up $\log \pi(y_w)$ and down $\log \pi(y_l)$.

Code

import torch
import torch.nn.functional as F


def seq_logp(logits, labels, mask):
    logp = torch.log_softmax(logits[:, :-1], dim=-1)
    tok = logp.gather(-1, labels[:, 1:].unsqueeze(-1)).squeeze(-1)
    return (tok * mask[:, 1:]).sum(-1)


def dpo_loss(pi_w, pi_l, ref_w, ref_l, beta=0.1):
    margin = beta * ((pi_w - ref_w) - (pi_l - ref_l))
    return -F.logsigmoid(margin).mean(), margin.detach()


loss, margin = dpo_loss(
    torch.tensor([-12.0]), torch.tensor([-15.0]),
    torch.tensor([-13.0]), torch.tensor([-14.0]),
)
print(loss.item(), margin.item())

This prints about 0.598 and 0.2. Precompute the reference log-probs once and cache them so you only keep one model on the GPU.

RLHF versus DPO

RLHF with PPODPO
Models in memorypolicy, reference, reward, valuepolicy, reference (precomputable)
Dataonline samples scored by a reward modelfixed preference pairs
Explorationyes: finds behaviour not in the datano: limited to the pairs
Stabilityfiddly, many hyperparameterssimple, one knob ($\beta$)
Failurereward hackinglikelihood displacement, length drift

Use DPO for style, tone and safety preferences when you have good pairs. Use online RL when you have a reward you can compute and want the model to discover new behaviour, as in the reasoning-model recipes next week.

Known failure modes

  • Reward hacking. The policy finds outputs the reward model likes and humans do not (longer, more confident, more flattering).
  • Likelihood displacement. DPO can lower the probability of both responses, because the loss only constrains their difference.
  • Verbosity drift. Preference data often favours longer answers. Check response length before and after.

Common mistakes

  • Summing log-probs over the prompt as well as the response.
  • Using a reference that is not the model you started from.
  • Setting $\beta$ too low, so the policy drifts far from the reference.
  • Judging success by the DPO loss. Look at held-out preference accuracy and read generations.

Check yourself

  • Why does $Z(x)$ cancel?
  • A pair has a scaled margin of $-0.4$ (already multiplied by $\beta$). What is the loss, and is its gradient weight larger or smaller than at margin $0.2$?
  • What does raising $\beta$ do to how far the policy may move?
  • Name one reason online RL can beat DPO on the same prompts.