II · THE IDEA · ARTIFICIAL INTELLIGENCE
DPO and Direct Preference Learning
▶ Listen · narrated
Training a language model to follow preferences once required building a separate reward model first. DPO showed you could optimise the language model directly, halving the pipeline and the compute budget.
At a glance
- What it replaces
- The reward-model stage in RLHF (reinforcement learning from human feedback)
- Training signal
- Pairs of answers to the same prompt, one preferred over the other
- Key insight
- The language model's own log probabilities can serve as an implicit reward
- Adoption
- Now widely used in instruction tuning, often under the umbrella term preference learning
Imagine teaching a dog two tricks: the old way, you first teach it to recognise a clicker sound as meaning "good", then use the clicker to teach the actual trick. DPO is more direct — you just reward the trick itself, skipping the clicker stage. For language models, the old RLHF method trained a separate model to score answers as good or bad, then used those scores to guide the language model. DPO skips the scoring model. It trains the language model directly on pairs of answers where humans said one was better, adjusting the model to make preferred answers more probable. The model's own internal numbers — the probabilities it assigns to each word — serve as the reward signal, so you do not need a second model to provide scores.
DPO reparameterises the RLHF objective to eliminate the reward model. In standard RLHF, you maximise expected reward from a learned reward function r(x,y) minus a KL penalty from a reference policy. DPO observes that the optimal policy under this objective has a closed form: π*(y|x) ∝ π_ref(y|x) exp(r(x,y)/β), where β controls the strength of the KL constraint. Rearranging, r(x,y) = β log(π*(y|x)/π_ref(y|x)) + const. You can substitute this into the Bradley-Terry preference model, which says P(y_w > y_l) = σ(r(y_w) - r(y_l)), giving a loss that depends only on policy and reference log probabilities, not on r explicitly. The loss is -log σ(β log(π_θ(y_w|x)/π_ref(y_w|x)) - β log(π_θ(y_l|x)/π_ref(y_l|x))). You optimise this with gradient descent directly on θ, the policy parameters. The reference model π_ref is frozen, typically initialised to the supervised fine-tuned model before preference training. Because the loss is differentiable and requires only forward passes through both models, training is stable and does not require sampling or value function approximation. The method cannot handle rewards that depend on multi-step interaction or external tool calls, because those cannot be expressed as functions of a single response's log probability.
Look closer
The training data is comparative, not absolute
You do not label answers as good or bad in isolation. Instead you present pairs: for this prompt, answer A was preferred to answer B. The model learns to increase the probability it assigns to A relative to B, but it never sees a numeric score. This is closer to how human raters actually work — it is easier to say which of two responses is better than to assign each a number on a scale — and it sidesteps the need to calibrate scores across different annotators or prompts.
The reference model stays frozen
DPO compares the model being trained against a frozen copy of itself from the start of preference training, usually called the reference model. This comparison prevents the model from collapsing into assigning all probability mass to a few preferred answers, a failure mode that would make it useless for general generation. The reference model acts as a regulariser, keeping the new model's behaviour tethered to the original distribution while nudging it toward preferred outputs. You pay for this in memory: both models must be loaded during training.
It is still supervised learning, not reinforcement learning
Despite replacing part of an RLHF pipeline, DPO does not use policy gradients, value functions, or any of the machinery of reinforcement learning. The loss function is differentiable and the gradients flow straight through the language model's existing architecture. This makes it simpler to implement and more stable to train than methods that treat text generation as a sequential decision problem, though it also means DPO cannot easily incorporate rewards that depend on multi-turn interaction or external tool use.
The story
The standard approach to aligning a language model with human preferences, documented in detail by OpenAI's InstructGPT work, involved two distinct stages after the initial pre-training. First, you trained a separate reward model: a classifier that learned to predict which of two answers a human would prefer, given a prompt. Then you used that reward model to guide the language model itself, typically through reinforcement learning, generating many candidate answers and adjusting the model's parameters to increase the reward scores it received.
This worked, but it was expensive. You needed a second model, often as large as the language model itself, and you needed to train it to convergence before you could begin the reinforcement learning stage. The RL stage was itself notoriously finicky, requiring careful tuning of hyperparameters to keep training stable. And because the reward model was a separate artifact, any mismatch between what it learned and what you actually wanted would propagate through the entire alignment process.
DPO, introduced by Rafailov and colleagues in 2023, observed that you could derive a closed-form expression relating the optimal language model to the optimal reward model. This meant you could rearrange the loss function to train the language model directly on preference pairs, without ever constructing the reward model explicitly. The language model's own log probabilities — the numbers it already computes when deciding which token to generate next — could serve as an implicit reward signal.
In practice, you start with a pre-trained model and a dataset of preference pairs: prompts where human raters have indicated that response A is better than response B. You also keep a frozen copy of the starting model as a reference point. For each pair, you compute how much more probable the policy model makes the preferred response compared to the rejected one, and how much more probable the reference model makes it. The loss function pushes these two ratios apart: the policy model should favour the preferred response more strongly than the reference model did, but not so strongly that it drifts into assigning near-zero probability to everything else.
The result is a single training loop with a straightforward supervised loss, no sampling of candidate responses during training, and no separate reward model to maintain. The method spread quickly after publication, and many openly released instruction-tuned models now list DPO or a close variant in their training recipe.
Why it mattered then
The paper appeared at a moment when RLHF had become the standard method for post-training alignment, but practitioners were acutely aware of its costs. Training a reward model required a substantial labeled dataset — InstructGPT reported using tens of thousands of comparisons — and the model itself was large, often comparable in size to the language model being aligned. The reinforcement learning stage added further expense: it required generating many samples per prompt, scoring them all with the reward model, and then running policy gradient updates, a process that was both slow and prone to instability. DPO offered a way out that did not require abandoning the preference-pair data format that had proven effective. It preserved the core insight that comparative judgments were more reliable than absolute scores, but eliminated the reward model entirely. For research groups without the budget to run full RLHF pipelines, or for practitioners tuning smaller models where the overhead of a separate reward model was proportionally larger, this was a meaningful reduction in complexity. The method also produced a simpler ablation story: if alignment quality changed, there was one fewer component to investigate.
Why it matters now
Direct preference optimisation and its variants have become a standard tool in post-training, appearing in the recipes for models released by academic labs, startups, and large companies alike. The term "preference learning" now often encompasses DPO, its extensions, and related methods that optimise directly on paired comparisons, and the approach has been adapted to settings beyond text, including image generation and code synthesis. The method's limitations have also become clearer with use. Because DPO trains on a static dataset of preference pairs, it cannot adapt to distribution shift during training the way an online RL method might. If the model's outputs drift into a region of response space that was not well covered by the original preference data, the training signal weakens. There is also growing evidence that the choice of reference model matters more than the original paper suggested: if the reference is too weak, the policy model may learn to exploit its failures rather than learning genuinely preferred behaviour. Several extensions have addressed these issues. Some methods periodically refresh the preference dataset by generating new responses from the current policy model and collecting new comparisons, creating a hybrid between DPO's simplicity and RLHF's adaptiveness. Others modify the loss function to handle preferences of varying strength, or to incorporate additional constraints like length penalties or safety filters. The core insight — that the language model's own probabilities can substitute for an explicit reward model — has proven robust enough to support this ongoing refinement.
The surprising detail
The DPO loss function can be derived by starting with the standard RLHF objective and performing a change of variables that eliminates the reward model. This means the two methods are optimising closely related objectives, not fundamentally different ones. In the limit of infinite data and perfect optimisation, they should converge to similar solutions. The practical differences — speed, stability, cost — stem from the training dynamics and the architecture of the pipeline, not from pursuing different goals. This also means that debates about whether DPO or RLHF produces better alignment are often actually debates about dataset quality, reference model choice, or hyperparameter tuning, not about the methods themselves.
Remember this
DPO turns preference pairs directly into model updates, skipping the reward model. Simpler and cheaper, but tied to a static dataset.
Test yourself
A team trains a model with DPO using preference data collected from the base model's outputs. After training, the model's responses have shifted substantially, and they want to continue improving it. Why might running a second round of DPO on the same preference dataset be less effective than the first round, and what would address this?
The preference pairs were collected by comparing outputs from the base model, so they cover the region of response space where that model operated. After DPO training, the model has moved: it now generates different kinds of responses, and the original preference data may not provide a strong signal for choosing between them. The annotations effectively become less relevant as the model's distribution shifts. To address this, you would generate a new batch of responses from the current model, collect fresh preference judgments on those responses, and train on the new pairs. This is sometimes called iterative DPO or online DPO, and it borrows the adaptive data collection strategy from online RL methods while keeping DPO's simpler training loop.
Go deeper
- Direct Preference Optimization: Your Language Model is Secretly a Reward Model · arXiv · Rafael Rafailov et al. · 2023-05-29
- Training language models to follow instructions with human feedback · arXiv · Long Ouyang et al. · 2022-03-04
Image: Original diagram, The Daily Triptych. Licence: Original work. Source.