# RLHF vs DPO

By @omar · Published Nov 2, 2026 · From the roadmap [Large Language Models, End to End](https://ornix.io/u/omar/large-language-models-end-to-end)

A pretrained model predicts text. An assistant does what you asked. Alignment is the step in between, and for a few years it meant one pipeline: RLHF. DPO showed that most of that pipeline can be collapsed into a single supervised-style loss. Knowing both is how you decide when each is the right tool.

## RLHF in three stages

1. **SFT.** Fine-tune on demonstrations of good behaviour. This gives a reference policy $\pi_{ref}$.
2. **Reward model.** Collect pairs where a human preferred response $y_w$ over $y_l$ for the same prompt $x$. Fit a scalar reward $r(x, y)$ with the Bradley-Terry loss:

$$\mathcal{L}_{RM} = -\log \sigma\big(r(x, y_w) - r(x, y_l)\big)$$

3. **RL.** Optimise the policy against the reward, with a leash so it cannot wander far from $\pi_{ref}$:

$$\max_{\pi}\ \mathbb{E}_{x,\ y \sim \pi}\big[r(x, y)\big] - \beta\, \mathrm{KL}\big(\pi \,\|\, \pi_{ref}\big)$$

PPO is the usual optimiser. In practice that means four large models in memory: the policy, the reference, the reward model and a value network (the critic).

## The DPO trick

The KL-regularised objective has a closed-form optimum:

$$\pi^*(y \mid x) = \frac{1}{Z(x)}\, \pi_{ref}(y \mid x)\, \exp\!\big(r(x, y)/\beta\big)$$

Rearrange to express the reward in terms of the policy:

$$r(x, y) = \beta \log \frac{\pi^*(y \mid x)}{\pi_{ref}(y \mid x)} + \beta \log Z(x)$$

Put that into the Bradley-Terry loss. The term $\beta \log Z(x)$ depends only on the prompt, so it cancels in the difference. What is left needs no reward model and no sampling:

$$\mathcal{L}_{DPO} = -\log \sigma\!\left(\beta\left[\log\frac{\pi(y_w \mid x)}{\pi_{ref}(y_w \mid x)} - \log\frac{\pi(y_l \mid x)}{\pi_{ref}(y_l \mid x)}\right]\right)$$

The model *is* the reward model: its implicit reward is $\beta \log(\pi/\pi_{ref})$.

```mermaid
%% ornix:id=rlhf0007
flowchart TB
    subgraph RLHF["RLHF (online)"]
        direction LR
        A1["SFT policy"] --> A2["Sample responses"]
        A2 --> A3["Reward model scores them"]
        A3 --> A4["PPO update with KL penalty"]
        A4 --> A2
    end
    subgraph DPO["DPO (offline)"]
        direction LR
        B1["SFT policy = reference"] --> B2["Fixed preference pairs"]
        B2 --> B3["Log-prob ratios vs reference"]
        B3 --> B4["Logistic loss, gradient step"]
    end
```

## A worked example

Sequence log-probabilities (sum over response tokens only), with $\beta = 0.1$:

| | policy $\log \pi$ | reference $\log \pi_{ref}$ | ratio |
|---|---|---|---|
| chosen $y_w$ | -12.0 | -13.0 | +1.0 |
| rejected $y_l$ | -15.0 | -14.0 | -1.0 |

Margin $= 0.1 \times (1.0 - (-1.0)) = 0.2$. Loss $= -\log \sigma(0.2) = 0.598$. At initialisation the policy equals the reference, the margin is 0 and the loss is $\ln 2 = 0.693$, so a healthy run starts at 0.693 and falls.

The gradient scales each pair by $\sigma(-\text{margin}) = 0.45$ here: pairs the implicit reward already ranks correctly get less weight, pairs it gets wrong get more. It pushes up $\log \pi(y_w)$ and down $\log \pi(y_l)$.

## Code

```python
import torch
import torch.nn.functional as F


def seq_logp(logits, labels, mask):
    logp = torch.log_softmax(logits[:, :-1], dim=-1)
    tok = logp.gather(-1, labels[:, 1:].unsqueeze(-1)).squeeze(-1)
    return (tok * mask[:, 1:]).sum(-1)


def dpo_loss(pi_w, pi_l, ref_w, ref_l, beta=0.1):
    margin = beta * ((pi_w - ref_w) - (pi_l - ref_l))
    return -F.logsigmoid(margin).mean(), margin.detach()


loss, margin = dpo_loss(
    torch.tensor([-12.0]), torch.tensor([-15.0]),
    torch.tensor([-13.0]), torch.tensor([-14.0]),
)
print(loss.item(), margin.item())
```

This prints about 0.598 and 0.2. Precompute the reference log-probs once and cache them so you only keep one model on the GPU.

## RLHF versus DPO

| | RLHF with PPO | DPO |
|---|---|---|
| Models in memory | policy, reference, reward, value | policy, reference (precomputable) |
| Data | online samples scored by a reward model | fixed preference pairs |
| Exploration | yes: finds behaviour not in the data | no: limited to the pairs |
| Stability | fiddly, many hyperparameters | simple, one knob ($\beta$) |
| Failure | reward hacking | likelihood displacement, length drift |

Use DPO for style, tone and safety preferences when you have good pairs. Use online RL when you have a reward you can compute and want the model to discover new behaviour, as in the reasoning-model recipes next week.

## Known failure modes

- **Reward hacking.** The policy finds outputs the reward model likes and humans do not (longer, more confident, more flattering).
- **Likelihood displacement.** DPO can lower the probability of *both* responses, because the loss only constrains their difference.
- **Verbosity drift.** Preference data often favours longer answers. Check response length before and after.

## Common mistakes

- Summing log-probs over the prompt as well as the response.
- Using a reference that is not the model you started from.
- Setting $\beta$ too low, so the policy drifts far from the reference.
- Judging success by the DPO loss. Look at held-out preference accuracy and read generations.

## Check yourself

- Why does $Z(x)$ cancel?
- A pair has a scaled margin of $-0.4$ (already multiplied by $\beta$). What is the loss, and is its gradient weight larger or smaller than at margin $0.2$?
- What does raising $\beta$ do to how far the policy may move?
- Name one reason online RL can beat DPO on the same prompts.

---

Roadmap: https://ornix.io/u/omar/large-language-models-end-to-end (Markdown: https://ornix.io/u/omar/large-language-models-end-to-end.md)

Canonical page: https://ornix.io/u/omar/large-language-models-end-to-end/rlhf-vs-dpo
