II · THE IDEA · ARTIFICIAL INTELLIGENCE
Preference Optimization with DPO
▶ Listen · narrated
Reinforcement learning from human feedback works, but it is brittle to run. Direct Preference Optimization keeps the same goal and drops the reward model and the policy-gradient loop entirely.
At a glance
- What it is
- A preference classification loss that aligns a language model without RL
- What it skips
- A separately trained reward model and a reinforcement-learning loop
- Core claim
- The language model already encodes an implicit reward
- Training data
- Pairs of preferred and dispreferred completions for the same prompt
- Nearby idea
- Self-play fine-tuning turns a model against its earlier copies
Think of teaching a cook with side-by-side plates rather than with a scorecard and a coach who shouts after every taste. You show two finished dishes for the same order and mark which one you prefer. Direct Preference Optimization is the version of alignment training that only needs those side-by-side marks.
Older pipelines first built a separate scoring model that tried to guess how much humans would like any dish, then used reinforcement learning to push the cook toward high scores. That works, but it is two systems to train and a fussy feedback loop. DPO uses maths that links the best cook to the scores they would have earned, and rewrites the problem so you only train the cook. If the preferred dish becomes more likely under the model than the other dish—compared with a frozen copy of the cook from earlier training—you are done with that example.
Self-play fine-tuning is a cousin: sometimes the “preferred” dish is simply whatever a later version of the model cooks when set against an earlier version, so the labels can come from the model’s own improvement path rather than only from outside judges.
DPO targets the same KL-constrained reward maximisation used in RLHF, but avoids fitting an explicit reward model r and avoids policy-gradient updates against it. Under a Bradley–Terry-style preference model, human preference probabilities depend on reward differences. The optimal policy π* for a reward r with KL penalty to a reference π_ref satisfies a relation that can be inverted: r can be expressed in terms of π* and π_ref (log-ratio of probabilities plus a partition term). Substituting that expression into the preference likelihood cancels the explicit reward and yields a loss on π alone.
The resulting objective is a logistic binary classification loss over pairs (x, y_w, y_l). It increases the relative log-probability of the winning completion y_w versus the losing completion y_l under π, scaled against π_ref, typically with a temperature β that plays the role of the KL strength. Training is offline on a fixed preference dataset; the reference is usually the SFT checkpoint and remains frozen.
Limitations track the assumptions: preference noise, composition of the pair distribution, and how far π moves from π_ref all affect stability and quality. DPO does not remove the need for good pairs; it removes the separate reward network and the RL loop. Self-play fine-tuning (e.g. methods that form winners and losers from later versus earlier iterates of the same model) likewise shapes a policy with comparative signal but obtains labels from the training dynamics rather than from the DPO closed-form reduction alone.
Look closer
The preference pair is the unit of supervision
Each training example is one prompt and two completions: one marked preferred, one dispreferred. The loss does not need absolute scores. It only needs the model to assign higher likelihood, relative to a frozen reference policy, to the preferred completion than to the dispreferred one. That relative comparison is enough to push the policy toward the preferred behaviour.
The reward model never appears as a separate network
Under a standard preference model, the optimal policy and the reward that produced it are linked by a closed-form relationship. DPO rearranges that relationship so the reward is an implicit function of the current policy and the reference policy. Optimising the preference likelihood therefore updates the generator directly. No second model is fitted, and no sampler has to maximise a learned scalar reward at training time.
Self-play is a different route to preference signal
Self-play fine-tuning builds preference-like pairs from the model itself: later iterates are treated as winners against earlier copies of the same model. The supervision still looks like a preference between two responses, but the labels come from the training dynamics rather than from an external annotator. It is a neighbour of DPO in spirit, not a drop-in description of the same algorithm.
The story
Aligning a language model with human preferences has often been framed as a reinforcement-learning problem. A common pipeline first fine-tunes the model on demonstrations, then trains a separate reward model on pairwise human comparisons, then updates the language model with a policy-gradient method so that it scores well under that reward while staying close to a reference policy. The goal is clear. The machinery is not light: reward models can be misspecified, and reinforcement learning on a moving language-model policy is sensitive to hyperparameters, reward scale, and KL penalties.
Direct Preference Optimization starts from the same formal objective and asks whether the detour through an explicit reward model and an RL optimiser is necessary. The answer it gives is that it is not. If preferences are modelled in the usual way, as a noisy comparison between the rewards of two completions, then the reward that is consistent with an optimal policy can be written in closed form in terms of that policy and a fixed reference policy. Substituting the expression back into the preference likelihood yields a loss that depends only on the policy being trained.
What remains is a binary classification problem on preference pairs. For each prompt, the model sees a preferred completion and a dispreferred one. The loss encourages the trained policy, measured in log-probability relative to the reference, to rank the preferred completion above the dispreferred one. Gradient steps on that loss move the language model itself. There is no separate network whose scalar output must be maximised, and no sampling loop that treats generation as an environment for policy gradients.
The reference policy matters. It is typically the model after supervised fine-tuning, held fixed. Deviations from it are penalised implicitly through the way the loss is written, which plays a role analogous to the KL constraint in RL-based preference methods. Without that anchor the policy could drift in ways that fit the pairwise labels while degrading fluency or coverage.
A related line of work, self-play fine-tuning, also builds training signal from comparisons between responses, but obtains those comparisons by pitting a model against earlier copies of itself rather than against human labels alone. The two ideas sit in the same broad family: preference-shaped updates that avoid a full reinforcement-learning stack. They are not the same algorithm, and the original DPO result is specifically about the closed-form reduction from reward-model RL to a classification loss on offline preference data.
In practice the method still inherits the limits of its data. Noisy, inconsistent, or narrowly sourced preference pairs will steer the model as surely as clean ones. The simplification is in the optimiser and the number of learned artefacts, not in the need for careful supervision.
Why it mattered then
When the DPO paper appeared, preference alignment was already central to making large language models usable, but the dominant recipe coupled a reward model to a reinforcement-learning stage that many groups found expensive and unstable to run. Showing that the same preference objective could be optimised with an offline classification loss lowered the barrier: fewer moving parts, no online sampling against a reward model during the RL phase, and a training setup closer to ordinary fine-tuning. That mattered immediately for labs that wanted alignment gains without maintaining a full RLHF stack.
Why it matters now
Preference optimisation remains how many instruction-tuned and assistant models are refined after supervised fine-tuning. DPO and its relatives are widely used because they fit existing fine-tuning infrastructure and keep the alignment stage inspectable as a loss on pairs. Understanding the reduction—why a language model can be treated as carrying an implicit reward—also clarifies what later variants are changing when they alter the reference, the sampling of pairs, or the loss, including self-play schemes that replace or augment human labels with comparisons against the model’s own earlier outputs.
The surprising detail
The title claim is literal: once the preference model and the KL-constrained objective are accepted, the language model is already a reward model in disguise. The reward for a completion can be read off from the log-probability ratio between the current policy and the reference policy, up to a normalisation term that does not depend on the completion’s absolute score in isolation. Alignment then becomes reparameterisation rather than a second learning problem bolted on top.
What is disputed
DPO’s derivation assumes a particular preference model and a KL-constrained form of the optimal policy. When real annotators disagree, when pairs are off-policy in problematic ways, or when later variants change sampling and regularisation, empirical behaviour can diverge from the clean closed form. Self-play fine-tuning is a related but distinct source of preference-like signal, not a restatement of DPO.
Remember this
DPO turns preference alignment into a classification loss on pairs by treating the language model’s own probabilities, relative to a reference, as an implicit reward.
Test yourself
A team has offline preference pairs and a supervised-fine-tuned checkpoint they can freeze as a reference. They are deciding between training a reward model and running RL, or running DPO. What exact piece of machinery does DPO let them drop, and what must still be true about their pairs for the loss to mean what they think it means?
They can drop the separately trained reward model and the reinforcement-learning loop that maximises its score; the policy is updated directly by a preference classification loss. The pairs must still be reliable relative labels for the same prompt—preferred versus dispreferred—because the method does not invent preference signal. It only reparameterises the usual preference objective so that fitting those labels updates the generator itself, anchored to the frozen reference.
Go deeper
- [2305.18290] Direct Preference Optimization: Your Language Model is Secretly a Reward Model · arxiv.org
- [2401.01335] Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Models · arxiv.org
Image: Original diagram, The Daily Triptych. Licence: Original work. Source.