II · THE IDEA · ARTIFICIAL INTELLIGENCE

Crowdsourcing High-Quality Human Feedback

data · InstructGPT; DPO · pairwise preferences · collection and aggregation

▶ Listen · narrated

A language model only learns what humans prefer if someone first collects those preferences, cleans them, and turns ranked outputs into something an optimiser can trust.

At a glance

Core unit: A prompt with two or more model completions humans can rank
InstructGPT path: Rankings train a reward model; PPO then optimises the policy
DPO path: Same pairwise preferences enter a direct policy loss
Label workforce: Screened contractors, not ad-hoc crowd clicks alone
Aggregation: Full rankings reduced to pairwise preferred–dispreferred pairs

Think of tasting two plates of food and saying which you prefer, not giving each a score out of ten. That is the basic unit of this work: for one question (the prompt), the model cooks several answers, and a person ranks them.

Those rankings are chopped into pairs—answer A beat answer B. Many such pairs teach a computer what “better” looks like. One approach builds a separate scoring machine from the pairs, then pushes the model to score highly. Another approach, DPO, uses the pairs to nudge the model straight away, without building that separate scorer.

The hard part is not the maths; it is the people and the rules. Who ranks, what instructions they follow, and how careless votes are removed decide whether the model learns a clear idea of helpful and safe behaviour or only a muddle of opinions.

Look closer

The comparison, not the score
Both pipelines begin from relative judgments. Labelers see several completions for one prompt and put them in order, or mark which of a pair is better. Absolute quality scores are avoided; the training signal is “A beats B on this prompt.” That choice keeps the task tractable for humans and matches the Bradley–Terry-style preference models that later stages assume.
Who is allowed to label
High-quality preference data is not gathered by opening an unfiltered public form. The InstructGPT work used hired contractors, screening, and written guidelines so that rankings reflected a coherent notion of helpful, honest, harmless behaviour rather than whatever a random clicker rewarded. Agreement between labelers, and between labelers and researchers, is imperfect; the methodology treats that noise as something to measure and manage, not something that disappears at scale.
One dataset, two optimisation stories
Once pairwise preferred and dispreferred completions exist, they can feed a separate reward model that a reinforcement-learning stage maximises, as in InstructGPT, or enter a classification-style objective that updates the policy directly, as in DPO. The collection and filtering problem is largely shared; the papers diverge on what happens after the preferences are clean enough to trust.

The story

Instruction-following models are only as aligned as the human judgments used to steer them. The practical problem is not merely “get some feedback,” but how to collect preference data that is consistent enough, large enough, and clean enough to train on—and how to aggregate noisy rankings into a signal an optimiser will not overfit or game.

The InstructGPT pipeline makes the data stages explicit. First, labelers write demonstration outputs for prompts; those demonstrations support supervised fine-tuning. Next, for a broader set of prompts, the model produces several candidate completions. Labelers rank those candidates. Rankings are decomposed into pairwise comparisons: whenever completion A is placed above completion B, the pair (prompt, A preferred, B dispreferred) becomes a training example for a reward model. That reward model is then the objective a policy optimises with reinforcement learning, using PPO. Every later stage inherits the strengths and defects of those ranked comparisons.

Collecting the rankings is a methodology problem before it is a modelling problem. Prompts must cover the behaviours one cares about. Completions must be diverse enough that rankings are informative, not ties between near-duplicates. Labelers need instructions that define what “better” means when helpfulness, truthfulness, and harmlessness pull in different directions. Screening and ongoing quality checks matter because a small fraction of careless or adversarial rankings can distort the reward model. Researchers also measure how often labelers agree with one another and with the research team; disagreement is expected, and the pipeline is designed around relative orderings rather than perfect consensus.

Aggregation is deliberately simple at the interface to learning. A full ranking of K outputs yields multiple pairwise preferred–dispreferred pairs. Those pairs train a reward model under a preference likelihood (in the spirit of Bradley–Terry): the model learns to assign higher scalar reward to the preferred completion than to the dispreferred one for the same prompt. Filtering can remove obviously broken outputs, near-duplicates, or comparisons where labelers could not decide. What remains is still human and noisy, but it is structured enough to supervise a reward head.

Direct Preference Optimization starts from essentially the same kind of pairwise data and asks a different question: if the goal is a policy that prefers A to B, must one train an explicit reward model and run PPO at all? DPO shows that the constrained reward-maximisation problem can be rearranged so the policy itself is updated with a classification-style loss on preferred versus dispreferred completions, using a reference model as an anchor. The collection problem does not vanish—someone must still produce reliable (prompt, chosen, rejected) triples—but the engineering path after aggregation shortens.

At scale, the scarce resource is not raw clicks but coherent preference judgments. Labeler selection, written criteria, ranking interfaces rather than absolute scores, and pairwise aggregation are the methodological spine shared by both lines of work. The algorithms differ; the demand for carefully gathered human rankings does not.

Why it mattered then

When InstructGPT was introduced, large language models were already fluent but unreliable at following user intent. Supervised demonstrations helped, yet they were expensive to write at volume and did not directly express trade-offs between competing answers. Ranked comparisons offered a way to harvest many judgments per prompt and to train a reward model that could generalise preference beyond the exact demonstration set. That mattered in the moment because it turned “alignment” into an operational data pipeline: hire and screen labelers, issue guidelines, collect rankings, fit a reward model, optimise with PPO. DPO later mattered for a complementary reason: teams already sitting on preference datasets needed a simpler training recipe that avoided the instability and implementation cost of RL while still using the same human comparisons. In both cases, the bottleneck that defined the moment was methodological—how to get preference data good enough that the subsequent optimiser was worth running.

Why it matters now

Open-weight fine-tuning has made preference optimisation widely accessible, but accessibility does not invent label quality. Anyone running DPO or a reward-model-plus-RL stack still depends on how prompts were chosen, how completions were sampled, who ranked them, and how ties, spam, and conflicting criteria were handled. Crowdsourcing without screening tends to reward superficial fluency; small, well-instructed label pools produce narrower but more coherent signals. As models are adapted for products, domains, and languages, the same design choices—pairwise rather than absolute scores, clear rubrics, aggregation into chosen/rejected pairs—decide whether the training signal reflects the intended users or only the loudest annotators. The papers remain relevant because they separate what must be true of the data from what can vary in the optimiser.

The surprising detail

The same ranked comparisons can train either an explicit reward model for PPO or a policy updated directly by DPO. The human methodology—screened labelers, multi-output rankings, decomposition into pairs—does more lasting work than the choice of which loss consumes the pairs. Preference quality is upstream of both algorithms; neither paper claims that scale alone repairs careless labeling.

What is disputed

Labeler agreement with researchers and with other labelers is imperfect in the reported InstructGPT work; preference data is inherently noisy. Neither paper claims a single best workforce size or ranking interface for every domain. DPO assumes preference data of the same pairwise form, but real-world gains still depend on how carefully those pairs were gathered and filtered.

Remember this

Preference learning is only as sound as the ranked comparisons you collect, who you allow to rank, and how you turn those rankings into preferred–dispreferred pairs.

Test yourself

You have full rankings of four completions per prompt from screened labelers. You plan to try both a reward-model-plus-PPO pipeline and DPO. What shared aggregation step should you perform once, and what does each method do with the result?

Go deeper

Image: Original diagram, The Daily Triptych. Licence: Original work. Source.

← Back to day 129