Direct Preference Optimization collapses the RLHF pipeline into one supervised loss. Here you train a real toy policy with the actual DPO objective, in your browser, and watch beta and step count decide how far it drifts from the reference model.
Sign in to save progress
One loss where RLHF needs a pipeline
Classic RLHF turns preference pairs into a trained reward model, then runs a reinforcement learning loop against it. If you have not walked that pipeline yet, the RLHF guide steps through all three phases. DPO starts from the same data and the same goal, then proves the whole right-hand side of that pipeline has a closed-form shortcut. It is why much of production preference tuning today skips the reward model.
RLHF
2 trained models, online1. Preference pairs
Humans compare completions: this one is better than that one. Same data both sides.
2. Train a reward model
A separate network learns a scalar score that imitates the human rankings.
3. RL loop (PPO)
Sample completions from the policy, score them with the reward model, update the policy under a KL penalty against the reference model. Repeat.
DPO
1 trained model, offline1. Preference pairs
The exact same human comparisons. No extra data is needed.
2. One supervised loss on the policy
No reward model: the policy's own log-probability ratios against the frozen reference play that role. No sampling: the dataset is fixed and training is plain gradient descent. Beta is the KL tether. The lab below runs this exact objective.
The reward model and the RL loop did not disappear. The DPO paper shows the RLHF optimum can be written in closed form, so both stages fold into the loss itself.
The objective, exactly as the lab runs it
loss(y_w, y_l) = -log sigmoid( beta * [ log( pi(y_w) / pi_ref(y_w) )
- log( pi(y_l) / pi_ref(y_l) ) ] )
implicit reward r(y) = beta * log( pi(y) / pi_ref(y) )
gradient weight sigmoid(-margin) wrong pairs push hardestWhy a reference model
The loss never scores probabilities directly, only log-ratios against a frozen copy of the starting model. That is RLHF's KL penalty smuggled into the objective: the policy is rewarded for moving relative to where it began, not for collapsing onto one answer.
What beta does
Beta multiplies the log-ratio margin inside the sigmoid, scaling the preference signal against the KL anchor. A high beta pushes harder per early step but saturates fast: small shifts already satisfy the pairs. A low beta pushes gently yet keeps drifting long after the ordering is learned, so final drift depends on beta and step count together.
Where the reward model went
r(y) = β·log(π/π_ref) is a valid reward model, and the DPO derivation shows the RLHF-optimal policy induces exactly this form. Your language model is secretly a reward model: train the policy and you get the rewards for free.
The lab: preference-tune a toy policy
The policy is a softmax over 8 completions of one fixed prompt, initialized to the reference model, which was authored to favor the vague and the confidently wrong answers. Every step button runs real gradient descent on the mean DPO loss over the six pairs. Every number on screen, probabilities, loss, KL, margins, is recomputed from the current logits. Train at least 10 steps, save the run, then do it again at a different beta.
The fixed prompt
"Explain overfitting in one sentence."
The preference data (all 6 pairs, authored once, never resampled)
Gradient descent on the real DPO loss
Gradient steps
0
DPO loss
0.6931
log 2 ≈ 0.6931 at step 0, for every beta
KL from reference
0.000
nats of drift
Current favorite
C
at 25.9%
Loss curve (every point recomputed from the logits)
Policy distribution over the 8 completions
Overfitting is when a model memorizes its training data instead of learning the pattern, so it fails on new data.
Overfitting: excess variance from minimizing empirical risk, harming generalization to the true distribution.
Overfitting is when a model does not work as well as you would want it to.
Overfitting is when a model is trained on too little data, and the fix is always adding more layers.
That depends on many factors and is difficult to explain in a single sentence.
Overfitting, which happens a lot, is basically when your model, after seeing the same examples many times, does great on those exact examples but then does much worse on anything new.
Your model crammed the answer key instead of studying the subject, so it flunks the real exam.
Gradient descent updates weights by following the negative gradient of the loss.
r is DPO's implicit reward, β·log(π/π_ref), recomputed from the current logits. Note H: it appears in no pair, so its logit receives exactly zero gradient, yet its probability still shrinks because softmax renormalizes as the winners grow.
Per-pair diagnostics
| pair | why | margin m | σ(m) satisfied | grad weight σ(−m) |
|---|---|---|---|---|
| A > C | Concise and correct beats vague filler. | 0.000 | 50.0% | 0.500 |
| A > D | Correct beats confidently wrong. | 0.000 | 50.0% | 0.500 |
| B > C | Jargon-heavy but accurate still beats saying nothing. | 0.000 | 50.0% | 0.500 |
| F > D | Rambling but correct beats a wrong answer. | 0.000 | 50.0% | 0.500 |
| A > F | Between two correct answers, concise wins. | 0.000 | 50.0% | 0.500 |
| G > E | A playful real answer beats an evasive non-answer. | 0.000 | 50.0% | 0.500 |
The gradient weight σ(−m) is DPO's built-in curriculum: pairs the policy currently gets wrong push hardest, and pairs it already satisfies fade toward zero. When every σ(m) passes 99%, training has effectively converged for this beta.
Your recorded runs
No runs saved yet. Train the policy above for at least 10 steps, then save the run to record its final loss, KL, and distribution here.