II · THE IDEA · ARTIFICIAL INTELLIGENCE
What Goes Into the Corpus
▶ Listen · narrated
The phrase trained on the internet hides months of decisions about what to keep, what to throw away, and how many times to show the model each surviving document.
At a glance
- Scale
- Modern pretraining corpora range from hundreds of gigabytes to multiple terabytes of text
- Sources
- Web crawls, books, code repositories, academic papers, sometimes social media and dialogue
- Deduplication
- Removing exact and near-duplicate documents improves downstream performance measurably
- Disclosure
- Major commercial labs now rarely publish corpus composition or filtering rules in detail
Imagine you are building a library to teach someone your language. You would not just photocopy every book you find. You would throw out duplicates, skip the ones that are mostly pictures or gibberish, and decide how many cookbooks to include compared to novels. A training corpus is that library. It starts with raw text from the web, books, code repositories and other sources, then filters out spam and low-quality writing, removes documents that appear multiple times, and sets rules for how often the model sees each type of text during training. The Pile, built in 2020, combined 22 datasets into 825 gigabytes and published every detail of how it was assembled. Most commercial corpora now keep those details secret, but the choices still happen: what to include, what to discard, and how much weight to give each source. Those choices determine what the model knows.
A training corpus is assembled in stages. First, raw data is collected from sources such as Common Crawl web archives, book corpora, GitHub repositories, academic paper databases and dialogue datasets. Second, filters are applied: minimum word count, maximum symbol-to-word ratio, language detection score, and often a learned quality classifier such as a fastText model trained on curated high- and low-quality examples. Third, deduplication removes exact duplicates by hashing and near-duplicates by identifying long shared substrings using suffix arrays or MinHash. Lee et al. showed that substring-level deduplication improves held-out perplexity and reduces memorisation. Fourth, the cleaned components are mixed according to sampling weights, which are hyperparameters set by the corpus authors. The Pile assigned each of its 22 components a weight; for example, books were upweighted relative to their byte share. Finally, the corpus is tokenised and sharded for distributed training. The tokeniser vocabulary is usually built from a subset of the corpus before filtering, which can create mismatches. Major labs no longer publish corpus composition, filtering heuristics or mixing weights, making it difficult to replicate results or audit what the model was exposed to. Licensing is also undisclosed: whether copyrighted books, paywalled articles or scraped social media appear in commercial corpora is not documented, and the legal status of such inclusion is contested in ongoing litigation.
Look closer
Deduplication is not just about exact copies
Katherine Lee and colleagues showed that removing documents similar at the substring level, not just byte-for-byte identical, makes models perform better on held-out evaluations and reduces the tendency to memorise training examples verbatim. The technique uses suffix arrays or MinHash to find long shared sequences efficiently across billions of documents. The Pile applied deduplication within each component dataset but not always across them, so a paragraph from a book republished on a website might appear twice under different labels.
Filtering rules are usually heuristic and underdocumented
Common filters remove documents below a minimum word count, above a maximum symbol-to-word ratio, or flagged by a separately trained quality classifier. Some labs filter by language detector score. Others remove lines that look like boilerplate or navigation elements. The Pile documented its per-source filters in detail; most commercial corpora do not. A heuristic that discards all documents containing certain substrings will silently erase discussions of those topics, and the model will never know they existed.
Mixing weights determine what the model sees most often
A corpus is not sampled uniformly. If books represent two per cent of the total byte count, the training loop might be configured to draw from books ten per cent of the time, upweighting that distribution. The Pile assigned each of its 22 components a sampling weight, published in the paper. Changing those weights changes the model's behaviour, sometimes dramatically, but the weights are a choice made by the people assembling the corpus, not a property of the data itself.
The story
Building a training corpus begins with acquisition. The Pile, assembled by EleutherAI and released in 2020, combined 22 existing datasets: a web crawl, several archives of books and academic papers, code from GitHub, dialogue from forums and subtitles, and specialist collections like patent text and medical abstracts. The result was 825 gigabytes of text, or roughly 300 billion tokens under the GPT-2 tokeniser. The paper listed every source, its size, and its sampling weight.
Before any of that text reaches the model, it passes through filters. Some are simple: discard documents shorter than a threshold, or with too many non-alphabetic characters. Others are learned: a fastText classifier trained on curated examples of high- and low-quality text scores each document, and anything below a cutoff is dropped. The filters are meant to remove spam, gibberish, navigation boilerplate and machine-generated dross, but they also encode assumptions about what good text looks like, and those assumptions are rarely tested against the full diversity of human writing.
Deduplication comes next. Katherine Lee and colleagues demonstrated in 2021 that removing near-duplicate documents improves model performance on downstream tasks and reduces memorisation of training data. Exact deduplication is straightforward: hash every document and discard repeats. Substring deduplication is harder. It looks for long sequences of tokens shared between documents, even if the documents as a whole differ. A suffix array or MinHash algorithm makes this tractable at scale. The Pile deduplicated within each component but not across them, so text appearing in both a book archive and a web crawl might survive twice.
Once filtered and deduplicated, the components are mixed. Mixing is not proportional to size. The Pile assigned each source a sampling weight: the model saw books more often than their share of the total byte count would suggest, and some niche sources less often. Those weights are hyperparameters, chosen by the corpus authors, and they shape what the model learns as much as the choice of sources does.
Most commercial labs no longer publish this level of detail. The datasets behind GPT-4, Claude and Gemini are not documented in released papers. We know they include web crawls, but not which domains were filtered out, how deduplication was configured, or how the mixing weights were set. Licensing is even less clear: whether the corpus includes copyrighted books, news articles behind paywalls, or scraped social media posts is usually not disclosed, and the legal questions remain unsettled in most jurisdictions.
Why it mattered then
The Pile was released in 2020 as a response to the lack of publicly available, well-documented pretraining corpora. GPT-2 had been trained on WebText, which OpenAI did not release. GPT-3 used an undisclosed mix. Researchers outside major labs could not replicate the results or study how corpus composition affected model behaviour, because they did not have access to comparable data. The Pile made 825 gigabytes available under permissive licences, with every source and every filtering step documented. It enabled EleutherAI to train the GPT-Neo and GPT-J models, and it became a standard benchmark dataset. The deduplication work by Lee and colleagues, published in 2021, provided empirical justification for a step that had previously been done inconsistently or skipped entirely. Together, these contributions established that corpus construction was not just data engineering but a research question with measurable consequences for model quality.
Why it matters now
Corpus composition is now one of the least transparent aspects of frontier model development. Labs cite competitive advantage and legal uncertainty as reasons not to disclose their data sources or filtering heuristics. This makes it difficult to audit what knowledge a model has access to, whose writing was included without consent, or whether filters have systematically excluded particular viewpoints or communities. The legal landscape is unsettled: ongoing lawsuits in the United States and Europe challenge whether scraping copyrighted text for training constitutes fair use or infringement, and the outcomes will determine whether current practices can continue. Meanwhile, researchers have shown that corpus details matter: models trained on different mixes of the same sources behave differently, and filters designed to remove low-quality text can also remove dialects, non-standard English, and writing by marginalised groups. Knowing what went into the corpus is a prerequisite for understanding these effects, but that knowledge is increasingly proprietary.
The surprising detail
The Pile included a component called Pile-CC, a filtered version of Common Crawl. Common Crawl is a nonprofit that archives the web and makes the raw data freely available, but the raw crawl is full of spam, duplicates and broken HTML. Pile-CC applied the same filters that had been used to create the C4 dataset, but the filters themselves were heuristics: discard pages with fewer than five sentences, or with a high ratio of non-alphabetic characters, or flagged as non-English by a language detector. Those heuristics were not designed by linguists or tested for bias; they were engineered to make GPT-2 perplexity go down. Yet they determined which parts of the web a generation of models would see. The Pile documented this openly, but most corpora built since have not.
What is disputed
The Pile documented its composition in detail, but it is not clear how representative its design choices are of other corpora. Most labs do not publish their filtering rules, deduplication thresholds or mixing weights, so we cannot know whether the techniques described here are widely used, modified in undisclosed ways, or replaced by other methods entirely. The legal status of including copyrighted material in training corpora is unresolved and differs by jurisdiction.
Remember this
A corpus is the result of hundreds of filtering, deduplication and weighting decisions, most of which are never published.
Test yourself
A lab builds two models with identical architectures and training procedures, but one is trained on a corpus where news articles are sampled at twice the rate of the other. Name two aspects of model behaviour that are likely to differ, and explain why corpus composition affects them even when the same articles appear in both corpora.
First, the model trained on more news will likely be better at tasks that resemble news writing: summarising events, naming public figures, and maintaining a formal register. It has seen more examples of that structure and vocabulary, so the patterns are more strongly weighted. Second, it may have a recency bias in its knowledge cutoff, because news articles trend toward recent events, and seeing them more often reinforces recent facts over older ones that appear mainly in books or archives. Corpus composition affects these even when both corpora contain the same articles because training is not about presence or absence, it is about frequency. A document seen once has far less influence on the learned weights than a document seen ten times. Sampling weights are not metadata; they are part of the training signal.
Go deeper
- The Pile: An 800GB Dataset of Diverse Text for Language Modeling · arXiv · Leo Gao et al. · 2020-12-31
- Deduplicating Training Data Makes Language Models Better · arXiv · Katherine Lee et al. · 2021-07-14
Image: Original diagram, The Daily Triptych. Licence: Original work. Source.