II · THE IDEA · ARTIFICIAL INTELLIGENCE
Human Feedback from Comparisons
▶ Listen · narrated
Absolute scores drift between raters and sessions. Pairwise choices are faster and more stable, and two foundational papers turned that fact into a practical alignment pipeline.
At a glance
- Core idea
- Fit a reward model to human choices between outputs, then optimise a policy against it
- Why pairs
- Relative judgements are more consistent than absolute numeric ratings
- 2017 paper
- Deep RL from human preferences on trajectory segments
- 2022 paper
- InstructGPT: SFT, reward model from rankings, then PPO
- Notable result
- 1.3B InstructGPT outputs preferred to 175B GPT-3
Think of grading essays. Asking “is this a 6 or a 7?” produces arguments between markers. Asking “which of these two essays is better?” is quicker and usually more consistent. The method here does the second thing.
Humans compare two behaviours or two model answers. A reward model learns to give a higher number to the one they picked. Reinforcement learning then adjusts the agent or language model so that its new outputs get higher numbers from that reward model. The 2017 work did this for agents in games and simulated robotics using short clips. The 2022 work did it for language models in three steps: imitate good demonstrations, learn a reward model from ranked answers, then optimise the model against that reward with PPO. That is how pairwise human feedback becomes a training signal.
Preference-based reward modelling fits a scalar rθ on inputs (trajectory segments in Christiano et al., prompt–completion pairs in Ouyang et al.) so that observed human pairwise orderings are likely under a comparison model—typically a Bradley–Terry-style loss on reward differences. Only relative order is supervised; absolute offset is unidentified.
Christiano et al. alternate collection of segment preferences, reward-model updates, and policy updates with deep RL, aiming for complex behaviour without a hand-specified reward and with relatively few labels.
Ouyang et al. instantiate RLHF for language models as: (1) supervised fine-tuning on labeler demonstrations; (2) reward-model training on rankings of multiple completions per prompt; (3) PPO on the SFT policy with the frozen reward model as reward, plus a KL-style penalty toward the SFT reference to limit drift. They report labeler preference for 1.3B InstructGPT outputs over 175B GPT-3 outputs on their evaluation setup.
Limitations inherent to the setup: reward-model error outside the preference distribution; policy exploitation of rθ (reward hacking); dependence on labeler pool and instructions; and the fact that PPO optimises the proxy, not live humans, at each gradient step.
Look closer
The comparison, not the score
Both lines of work treat the human as a source of pairwise orderings rather than calibrated numbers. In the 2017 setting a rater sees two short trajectory segments and picks which is better; in the 2022 language-model setting a rater ranks several model completions for the same prompt. The training target for the reward model is to assign higher scalar reward to the preferred item, so the loss is defined on differences, not on matching an absolute scale that raters do not share.
A frozen reward, a moving policy
Once the reward model is trained on the collected comparisons, reinforcement learning updates the policy to raise the reward model's score. The 2022 pipeline uses PPO for that step after a supervised fine-tune on demonstrations. The reward model is held fixed during policy optimisation; if the policy drifts into regions where the reward model was never trained, the scalar it returns can become a poor proxy for the original human preference.
Scale of the preference win
Ouyang et al. report that labelers preferred outputs from a 1.3 billion parameter InstructGPT model to outputs from a 175 billion parameter GPT-3 model. The comparison is not a claim that smaller models are universally better; it is evidence that preference-tuned behaviour on the evaluated prompts outweighed raw pre-training scale for those raters and tasks.
The story
Human feedback enters modern language-model alignment less often as a grade out of ten and more often as a choice: this completion rather than that one. The reason is practical. Absolute ratings wander with mood, scale interpretation and fatigue. A forced choice between two concrete outputs is faster to give and tends to agree better across people and across days. That relative signal is enough to train a reward model, and the reward model is enough to drive a policy update.
The 2017 paper by Christiano and colleagues stated the loop in a reinforcement-learning setting that was not yet about large language models. An agent produces behaviour; humans compare short trajectory segments; a reward model is fitted so that preferred segments score higher; the agent is then trained with reinforcement learning on that learned reward. The point was to obtain usable reward signal in domains where a hand-written reward function is hard to specify, using far fewer human judgements than would be needed if every state were labelled.
The 2022 InstructGPT paper adapted the same preference idea to instruction-following language models and made the pipeline explicit in three stages. First, labelers write demonstrations of desired behaviour and the base model is supervised-fine-tuned on them. Second, labelers rank model outputs for the same prompt; those rankings train a reward model. Third, the fine-tuned model is further optimised with PPO so that its outputs score highly under the frozen reward model, while a penalty keeps the policy from drifting too far from the supervised model.
What the two papers share is the editorial claim of this lesson: pairwise human preference is treated as a more reliable training signal than absolute scores. The 2017 work showed the loop could teach complex behaviour from segment comparisons. The 2022 work showed that the same preference machinery, applied after supervised fine-tuning, could make a much smaller model’s outputs preferred to those of a far larger untuned model on the instructions the labelers cared about.
Neither paper claims that preference data solves alignment in full. The reward model only sees the comparisons it was given; the policy only maximises what that model scores. Where raters disagree, where prompts fall outside the collected distribution, or where the policy finds loopholes in the learned reward, the signal can still mislead. The method’s strength is narrower and more concrete: for the costly step of turning human judgement into a scalar the optimiser can climb, comparisons beat absolute ratings.
Why it mattered then
Before these pipelines, specifying reward for rich behaviour meant hand-written functions or scarce absolute labels. Christiano et al. showed that segment comparisons could stand in for an explicit reward in deep RL, at a human-labelling cost low enough to be practical. Ouyang et al. then showed that the same preference structure could sit between a pre-trained language model and an instruction-following policy, at a moment when raw scale alone was not producing reliable compliance with user intent. In both moments the method mattered because it made human judgement legible to gradient-based optimisers without requiring raters to share a numeric scale.
Why it matters now
Preference-trained reward models remain a central piece of how large language models are aligned to follow instructions and to avoid disallowed behaviour. Understanding that the supervisory signal is relative—pair A beats pair B—clarifies both the power and the failure modes of current systems: reward hacking, distribution shift away from the comparison data, and the fact that “better according to the reward model” is only as good as the preferences collected. Anyone evaluating or fine-tuning open models inherits this design choice whether or not they retrain the reward model themselves.
The surprising detail
Labelers preferred 1.3B InstructGPT outputs to those from 175B GPT-3—the preference win came from the feedback pipeline, not from parameter count. The smaller aligned model beat a model more than a hundred times larger on the judged outputs, which undercuts the intuition that scale alone buys instruction-following.
What is disputed
The claim that pairwise judgements are more reliable than absolute ratings is the editorial premise supported by how both papers design collection and loss; the sources do not publish a single head-to-head psychometric trial that settles every domain. Preference wins are also specific to the labelers and prompt distributions used in evaluation.
Remember this
Pairwise preferences train the reward model; the policy then climbs that learned reward. Comparisons, not absolute scores, are the human signal.
Test yourself
A team replaces pairwise rankings with a 1–7 quality score per completion. Name two distinct risks this reintroduces for reward-model training, relative to the comparison-based approach in the papers.
Raters may not share a calibrated scale, so the same perceived quality becomes different numbers across people and sessions, adding noise the reward model must fit. Absolute scores also invite drift over time as anchors shift, whereas a forced choice only asks which of two visible outputs is better—the signal the pairwise loss expects.
Go deeper
- [1706.03741] Deep reinforcement learning from human preferences · arxiv.org
- [2203.02155] Training language models to follow instructions with human feedback · arxiv.org
Image: Original diagram, The Daily Triptych. Licence: Original work. Source.