Cleaning Web-Scale Training Data

Web-scale training data does not arrive as a curated library. It arrives as a crawl — an automated sweep that saves billions of pages exactly as it finds them. Mixed in with the writing are navigation bars, spam, machine-generated filler, mirrored sites, and the same article saved under many different addresses. Two lines of work turn that pile into something a language model can learn from. The first is quality filtering: throwing out pages that fail simple tests. The second is deduplication: removing copies. The Colossal Clean Crawled Corpus, known as C4, was built for the T5 transfer-learning study and shows the first. Lee and colleagues studied the second directly.

C4 begins with Common Crawl, a large public archive of web pages, and applies automatic rules rather than human judgement. A language detector — a program that guesses which language a page is written in — keeps the English pages and drops the rest. A line filter deletes any line that does not end in a full stop, question mark or similar closing punctuation. The reasoning is mechanical: real sentences usually end that way, and menu items and stray fragments usually do not. Pages holding fewer than a set number of sentences go too. A blocklist of banned words and a blacklist of banned domains remove pages tied to offensive or adult material. Pages that are mostly placeholder filler text, or mostly Javascript source code rather than prose, are discarded. Finally, any three-sentence span that appears more than once in the corpus is kept only once. What survives is still web text, but with much of the obvious rubbish gone.

These rules are crude on purpose. A line without a full stop may be a heading or a line of poetry. A page caught by the blocklist may be news reporting or fiction. The designers accepted those mistakes because every rule has to run across an enormous archive, so only cheap tests are affordable. The consequence is that the filter's habits — a preference for English, for conventional punctuation, for prose that looks like an article — pass into every model trained on the result.

Deduplication fixes a different problem. When the same page appears twice in the training set, the model is taught the same words twice, and that second pass buys nothing new. Near-duplicates do the same damage less visibly: a mirror with a few words changed, a template page with only the date swapped, a syndicated story republished on ten sites. Lee and colleagues found that a large share of web-derived training text is redundant in this way, and that long identical passages also turn up inside documents that are otherwise quite different.

Their pipeline works in layers. Identical documents are caught by hashing: each document is reduced to a short fingerprint, and matching fingerprints mean matching text. Near-duplicates are caught by MinHash, which builds a compact sketch from each document's overlapping word sequences, so that two texts sharing most of their phrasing produce similar sketches and can be merged even though no exact match exists. A final pass hunts for long stretches of text repeated across documents — footers, syndicated paragraphs, boilerplate — and strips those.

The reported effects are practical. For a fixed amount of computing, models trained on deduplicated data do better, because their capacity goes on varied examples instead of repeats. Memorisation drops: when an unusual passage appears once rather than dozens of times, the model is far less likely to reproduce it word for word when prompted. And when the training set and the evaluation set are deduplicated against each other, the test scores measure something real. Otherwise a model can score well simply because the test text was sitting in its training data under another address.

So filtering and deduplication sit upstream of every argument about model architecture. The same transformer will pick up different habits from a crawl still stuffed with copies of one FAQ than from a crawl stripped and uniqued. Neither step is neutral and neither is finished: toxicity filters cannot read context, quality rules bin odd but genuine writing, and approximate matching both merges texts it should not and misses copies it should catch. Skip them, though, and the training run is mostly a study of the web's own repetition and debris.