II · THE IDEA · ARTIFICIAL INTELLIGENCE
Web-Scale Text Corpora Curation
▶ Listen · narrated
A language model trained on the same paragraph a thousand times does not become a thousand times wiser. It becomes more likely to spit that paragraph back out.
At a glance
- Core problem
- Web crawls contain exact and near-duplicate passages at large scale
- Main lever
- Remove duplicate documents and long repeated substrings before training
- Observed effect
- Deduplicated data yields stronger models that memorise training text less
- Two techniques
- Near-duplicate document matching, and exact repeated-substring removal
Think of a library that somehow shelved the same photocopied essay hundreds of times, mixed in with unique books. A student who studied only by walking the shelves would reread that essay far more often than anything else, and might quote it from memory even when asked a different question.
Web text is like that library. Crawlers collect pages; many are mirrors, reposts, or pages that share long identical blocks. Filtering throws out obvious junk. Deduplication tries to keep one copy of near-identical pages and to strip long repeated passages. Quality scoring ranks what is left when there is still too much to keep.
After that cleaning, models generally do better on fresh questions and are less prone to spit out training text word for word. The crawl only becomes a training set once someone has decided what not to show the model again and again.
Web-scale corpus curation sits in front of tokenisation and training. Typical stages include language and length filters, heuristic or model-based quality scoring, near-duplicate document clustering, and exact repeated-substring removal.
Near-duplicate detection usually hashes or sketches each document (for example with MinHash-style fingerprints over shingles) so that pages with high overlap collide in a similarity index; one representative per cluster is retained. Exact substring deduplication finds long token or character spans that recur across the corpus and removes or masks redundant occurrences, which document-level clustering misses when the surrounding page differs.
Empirically, training on deduplicated data has been shown to improve evaluation performance relative to training on the same raw crawl, and to reduce emission of memorised training strings. A related step is cross-split deduplication: dropping or flagging spans that appear in both train and evaluation sets so that benchmarks measure more than reconstructive recall.
Limitations are real. Sketch thresholds trade false merges against missed duplicates. Aggressive substring removal can erase legitimate repeated forms (standard headers, fixed legal phrasing that is the object of study). Quality classifiers import the biases of whatever data or raters defined “good”. Curation parameters therefore need to be treated as part of the experimental record, not as invisible defaults.
Look closer
Duplicates are not only whole pages
Curation has to catch two different shapes of repetition. Entire documents can be near-copies of one another — mirrored sites, scraped reprints, boilerplate frames with a few words changed. Separately, long identical substrings can recur inside otherwise distinct pages: licence blocks, navigation chrome, template sentences, or the same paragraph copied across many articles. A pipeline that only collapses identical URLs still leaves both kinds of waste in the stream.
Near-duplicate detection is approximate on purpose
Exact string equality is too brittle for the open web, where a footer, timestamp, or tracking parameter can make two pages look different while their substance is the same. Near-duplicate methods therefore compare compact sketches of documents rather than full text, so that pages which largely overlap can be grouped and reduced to a single representative. The trade-off is tunable: too strict and true copies survive; too loose and genuinely distinct writing is discarded.
Train and test can silently share text
When the same passage appears in both the training split and an evaluation set, reported gains partly measure regurgitation rather than generalisation. Deduplicating across those boundaries is a curation step with direct consequences for how honest a benchmark score is. The contamination is often invisible until someone searches for long matching spans, not single words.
The story
Web-scale language-model training begins with text that nobody wrote for that purpose. Pages are crawled, stripped of markup, and poured into enormous corpora. Along the way the same material arrives many times: site mirrors, syndicated articles, terms-of-service blocks, and forum posts copied between threads. Left alone, those repetitions become a substantial fraction of what the model sees.
Curation is the work of deciding what survives. Filtering removes pages that fail crude quality checks — wrong language, too short, too much boilerplate, or other heuristic red flags. Quality scoring tries to rank what remains so that cleaner or more informative text is kept preferentially when budgets force a cut. Deduplication attacks repetition directly.
One line of attack targets whole documents that are near copies of each other. Rather than comparing every pair of pages in full, systems build short fingerprints of each document and group those whose fingerprints collide. From each group a single copy is kept. The method is deliberately approximate: it is meant to catch pages that differ only in minor ways while still running over corpora too large for naive comparison.
A second line of attack targets long identical substrings that recur inside different documents. Even after near-duplicate pages are collapsed, the same paragraph can still appear thousands of times as a quoted block, a template, or a legal notice. Finding those repeated spans and stripping or down-weighting them reduces the chance that training will hammer the same token sequence over and over.
Work on this problem has shown that the effects are not cosmetic. Models trained on deduplicated data tend to perform better on held-out evaluation, and they emit memorised training text less readily. That second point matters for more than cleanliness of metrics: verbatim regurgitation of training data is one of the clearer ways a model can leak private or copyrighted material that happened to sit in the crawl.
The same discipline applies across the train–test boundary. If a benchmark passage also lives in the training set, a high score may partly reflect recall of that passage rather than broader competence. Curation that searches for overlapping spans between splits is therefore part of making reported progress trustworthy.
None of this is a single finished recipe. Filtering heuristics disagree about what “quality” means; aggressive deduplication can discard useful paraphrase; quality scorers trained on one distribution can quietly favour another. The underlying claim is simpler: at web scale, the raw crawl is not a training set until someone has decided what to drop, what to keep once, and what never to score against.
Why it mattered then
As language models moved onto web-scraped corpora, the assumption that more raw tokens were always better began to fray. Duplication was not a minor archival nuisance; it warped the effective training distribution, inflated apparent dataset size, and made memorisation easier to demonstrate. Systematic deduplication offered a concrete intervention: the same model family, trained with and without the repeated material, could be compared directly on quality and on how often training text reappeared in outputs. That framed curation as a first-class part of the training recipe rather than a preprocessing afterthought.
Why it matters now
Open crawls and open-weight training runs still start from text that is messy, mirrored, and unevenly valued. Anyone assembling a corpus faces the same choices about filters, near-duplicate thresholds, substring removal, and train–test overlap. Those choices affect benchmark honesty, training cost, and how much verbatim text the finished model is liable to reproduce. Deduplication and quality scoring remain among the few levers that improve models without changing architecture or adding parameters.
The surprising detail
Deduplication can improve a model while also making it less likely to quote its training data. The two results travel together: repetition in the corpus both wastes capacity on redundant gradient updates and teaches the model that certain long strings are extremely likely. Removing those strings is simultaneously a quality intervention and a memorisation intervention — not two separate campaigns.
What is disputed
Reported gains from deduplication depend on the corpus, the duplicate-detection thresholds, and the model scale under test. Exact fractions of duplicated content and exact downstream lifts are not universal constants; they are measurements on particular crawls and pipelines. Filtering and quality-scoring heuristics are still more disputed than exact substring deduplication, because “quality” is not a single observable.
Remember this
At web scale, repetition is a training signal. Deduplicate documents and long substrings, or the model will learn the copies.
Test yourself
A team collapses near-duplicate web pages into single representatives but does not remove long identical substrings that still appear across otherwise different pages. What failure mode remains, and why can it affect both model quality and evaluation?
Template paragraphs, licence blocks, and other repeated spans still reach the trainer many times, so the model over-fits those sequences and may regurgitate them. If the same spans also sit in evaluation data, benchmark scores partly measure memorisation of shared text rather than generalisation. Document-level near-deduplication alone does not catch this.
Go deeper
- [1911.10600] Invenio: Discovering Hidden Relationships Between Tasks/Domains Using Structured Meta Learning · arxiv.org
- [2107.06499] Deduplicating Training Data Makes Language Models Better · arxiv.org
Image: Original diagram, The Daily Triptych. Licence: Original work. Source.