Data Selection via Perplexity for Language Model Training

Raw web text is abundant and uneven. Some passages are carefully written prose; many more are menus, cookie notices, duplicated product listings, or the same article scraped from a dozen mirrors. A language model will spend gradient updates on whatever it is shown. If the stream is full of repeats and low-value domains, those updates are not free: they cost compute, they shape what the model memorises, and they can distort evaluation when test documents are cousins of training documents.

One response is to clean the corpus before any large training run begins. Deduplication work from 2021 makes the case in plain terms. Exact duplicate removal is the easy end of the spectrum; near-duplicate detection catches pages that differ by a banner, a timestamp, or a thin rewrite. After such filtering, models tend to memorise less verbatim text and to need fewer steps to reach a given level of language-modelling performance. Overlap between train and evaluation data also falls, which means reported gains are less likely to be artefacts of having already seen the test distribution in slightly altered form. The method is unglamorous — hashing, similarity thresholds, held-out checks — but it changes the dataset the optimiser actually sees.

Cleaning still leaves a second problem. Even a deduplicated corpus is a mixture of domains that do not contribute equally to downstream usefulness or to the losses the trainer cares about. Uniform sampling by source size is a policy, not a law of nature. It may over-serve domains that are easy or huge, and under-serve domains that remain hard for the model yet matter for the tasks people care about later.

DoReMi attacks that mixture problem with a proxy. Instead of guessing domain weights by hand, it trains a smaller model under a minimax-style objective: group distributionally robust optimisation over domains, using excess loss relative to a reference so that the procedure does not simply pour all mass into intrinsically high-entropy domains. The output is not a new architecture. It is a set of sampling weights. Those weights are then used when a larger model is pretrained. In the reported experiments, the large model reaches target perplexities in fewer steps than a baseline mixture, and downstream evaluations improve under the reweighted data. The proxy is doing the expensive search over how much of each domain to keep in the stream; the main run mostly consumes the answer.

Perplexity and training loss sit near the centre of both stories even when the paper titles do not say “filter every document by a single perplexity cutoff.” A language model’s loss on a span is a direct score of how surprising that span is under the model. Deduplication removes mass that would otherwise be predicted with artificially high confidence because it was seen in near-identical form. Mixture optimisation reallocates mass toward domains whose excess loss says they are still under-served. In practice, teams often combine families of tricks: drop trash and duplicates, score documents with a small model or a classifier distilled from human preference, then set domain weights with something like DoReMi or with simpler heuristics. The shared pattern is that data is no longer treated as a fixed pile. It is ranked and budgeted.

None of this makes the large model “understand quality” in a human sense. The proxy only knows what its own training signal defines as costly. If that signal is misaligned with the evaluation you care about, the selected mixture can be confidently wrong. If near-deduplication thresholds are too aggressive, rare but legitimate variants disappear. If domain labels are coarse, weight optimisation moves blunt blocks of the internet rather than fine slices. The papers are clear that the gains are empirical and setup-dependent, not a universal proof that every corpus should be carved the same way.

What they do establish is a shift in habit. At web scale, the bottleneck is not only how many parameters you can afford. It is whether each token in the stream deserves its place. A small model, used either to help strip redundancy or to rebalance domains, is one of the few tools cheap enough to run across the pile before the main training bill begins.