II · THE IDEA · ARTIFICIAL INTELLIGENCE
Pretraining
▶ Listen · narrated
The task is mundane—predict the next token—but done over hundreds of billions of words, it produces behaviour no one programmed in.
At a glance
- What it is
- Training a model to predict the next token, over a very large corpus, before any task-specific tuning
- Typical corpus
- Hundreds of gigabytes to terabytes of text from books, web pages, code, academic papers
- Duration
- Weeks to months of continuous compute on hundreds or thousands of accelerators
- The objective
- Minimise cross-entropy loss on next-token prediction across the entire corpus
Imagine you are learning a language by reading everything you can find—novels, instruction manuals, poetry, code, legal documents—and after each sentence, you try to guess the next word. When you guess wrong, you adjust your mental model slightly. When you guess right, you gain confidence in that pattern. You do this billions of times, and eventually you have compressed so much of the language's structure into your head that you can write in styles you have never explicitly practised, translate between languages without a dictionary, even reason through problems, because all of those skills help you predict what comes next. Pretraining is that process, but done by a model with billions of parameters, reading hundreds of billions of words, adjusting its weights after every guess. The task is trivial—predict the next token—but the scale is not, and the scale is what produces behaviour that looks intelligent.
Pretraining is unsupervised learning via next-token prediction over a large, diverse text corpus. The model is initialised with random weights and trained to minimise cross-entropy loss: given a sequence of tokens, predict the probability distribution over the vocabulary for the next token, then compare that distribution to the actual next token and compute the loss. Gradients are backpropagated, weights are updated via an optimiser such as AdamW, and the process repeats across billions of training steps. The corpus is typically a mixture of sources, each weighted according to a sampling schedule. The Pile, for instance, combines twenty-two datasets with different sampling rates, so that smaller high-quality sources are not drowned out by larger low-quality ones. Training runs for multiple epochs over the data, though some sources are upsampled or downsampled to control their effective contribution. The resulting model has no explicit task-specific knowledge—there are no labels for translation, summarisation, or question-answering—but it has learned statistical patterns that generalise to those tasks. Few-shot learning, where the model adapts to a new task from a few examples in the prompt, emerges because the model has seen analogous patterns in the pretraining data and can apply them in new contexts. The compute cost scales roughly as the product of model size, corpus size, and training duration, and for frontier models this can exceed tens of thousands of petaflop-days. Pretraining is therefore done once, and the resulting model is then fine-tuned or aligned for specific applications.
Look closer
The objective never changes
Every update to the model's weights is driven by the same question: given this sequence of tokens, what should come next? There are no separate objectives for translation, summarisation, reasoning or code. Those behaviours emerge as side effects of compression—patterns the model learns because they help it predict held-out text more accurately. The pretraining loop is therefore extraordinarily simple: sample a chunk of text, mask out the tokens the model must predict, compute the loss, adjust the weights, repeat.
The corpus is assembled, not authored
The Pile, used to train several notable models, contains twenty-two distinct sources: books, patents, PubMed abstracts, GitHub repositories, philosophy papers, Ubuntu IRC logs, YouTube subtitles. Each source is weighted, and those weights are design decisions—more code, less chat; more formal prose, fewer forum threads. The result is not a neutral sample of all text, but a deliberate mixture intended to produce a model with particular strengths. Different corpora produce models with measurably different behaviour, even when the architecture and scale are held constant.
The cost is dominated by time and electricity
GPT-3's pretraining reportedly required thousands of petaflop-days of compute. Translating that into more tangible units: months of wall-clock time on a cluster of high-end accelerators, consuming megawatts continuously. The electricity cost alone runs into millions of dollars, and the hardware depreciation adds millions more. Pretraining is therefore done once, by organisations with substantial capital, and the resulting model is then fine-tuned or adapted for specific uses. You do not restart pretraining because you want the model to be polite; you do it because you are building a new foundation.
The story
Pretraining is the first and longest phase of training a large language model. The model begins with random weights—parameters initialised to small values with no structure. It is then shown vast quantities of text, one chunk at a time, and asked to predict the next token. When it predicts badly, the loss is high, and the weights are adjusted slightly to make that particular prediction better next time. When it predicts well, the loss is low, and the adjustment is smaller. This loop runs billions of times.
The corpus is not a single document. It is a mixture of sources, each contributing a portion of the total data budget. The Pile, for instance, combines twenty-two datasets: books from Project Gutenberg, academic papers from arXiv and PubMed, code from GitHub, legal opinions, patents, web pages, even transcripts of YouTube videos. Each source is weighted, and the weights matter. A model trained on more code will be better at code; a model trained on more formal prose will produce more formal prose. The mixture is a design choice, not a given.
The scale is difficult to convey in familiar terms. GPT-3 was trained on three hundred billion tokens. If you read at two hundred words per minute, eight hours a day, without pause, it would take you roughly four hundred years to read that much text. The model sees it all in a matter of weeks, because it is processed in parallel across thousands of accelerators. Each one performs trillions of floating-point operations per second, and the operations are almost entirely matrix multiplications—the mathematical backbone of the transformer architecture.
Nothing in the training loop explicitly teaches the model to translate, answer questions, or write code. Those abilities emerge because they compress the data. If the model learns that "chien" often appears in French text where "dog" appears in English text, and if it learns the grammatical patterns of both languages, it can use that knowledge to predict held-out French sentences more accurately. Translation becomes a side effect of compression. The same applies to reasoning: if the model learns that certain argument structures predict certain conclusions, it can apply those structures in new contexts, because doing so reduces its prediction error.
The process is unsupervised. There are no labels marking which sentences are translations, which are summaries, which demonstrate reasoning. The model infers structure from the statistical patterns in the text itself. This is why pretraining is sometimes called self-supervised: the supervision signal comes from the data, not from human annotation. The next token is always there, waiting to be predicted, and the cumulative effect of billions of predictions is a model that has compressed a substantial portion of human written knowledge into its weights.
Why it mattered then
Pretraining at this scale became possible only when three things converged: transformer architectures that could parallelise across very long sequences, hardware accelerators capable of sustained petaflop performance, and datasets large enough to justify the compute. The Pile was released in 2020 specifically to address the third constraint—most earlier corpora were either proprietary or too small to train models beyond a certain size. GPT-3, trained the same year, demonstrated that scale alone could produce qualitatively new behaviour. Few-shot learning, where the model adapts to a new task from a handful of examples in the prompt, was not designed into the architecture; it emerged from pretraining on enough diverse text that the model had seen analogous patterns thousands of times before. The result shifted the economics of model development. Instead of training a new model for each task, organisations could pretrain once and adapt the resulting model to many tasks with far less compute.
Why it matters now
Pretraining remains the most expensive phase of building a large language model, and the cost is rising. Models trained in 2024 reportedly use corpora exceeding a trillion tokens, and the compute budgets are proportionally larger. This creates a consolidation pressure: only a few organisations can afford to pretrain at the frontier, and everyone else fine-tunes their models or uses them via API. The composition of the pretraining corpus also matters more as models grow. Biases in the data—overrepresentation of English, underrepresentation of non-Western perspectives, inclusion of toxic or outdated material—are baked into the weights during pretraining and are difficult to remove afterwards. Some of the capabilities that make models useful, such as the ability to follow instructions or refuse harmful requests, are added in later phases, but the foundation is set here. Pretraining is therefore not just an engineering challenge; it is an editorial one, with consequences that persist through the model's entire lifecycle.
The surprising detail
The Pile contains a component called Pile-CC, derived from Common Crawl snapshots of the web. It is by far the largest single source, contributing roughly a quarter of the total data. But Common Crawl is famously noisy—it includes spam, broken pages, machine-generated text, duplicate content. The Pile's authors applied filters to remove the worst of it, but the filters are heuristics, not guarantees. A model trained on the Pile has therefore seen a substantial volume of text that no human would consider worth reading, let alone worth learning from. Yet this noise does not seem to degrade performance as much as one might expect. The prevailing hypothesis is that the sheer scale of the clean data drowns out the noise, and the model learns to assign low probability to incoherent patterns. But this is conjecture, not settled science. No one has run the experiment with a perfectly curated corpus at the same scale, because curating a trillion tokens by hand is not feasible.
Remember this
Pretraining is compression at scale. The mundane objective—predict the next token—produces emergent behaviour when applied to enough text.
Test yourself
Why does increasing the size of the pretraining corpus tend to improve few-shot learning, even though few-shot learning is not an explicit training objective?
Few-shot learning depends on the model recognising patterns in the prompt that it has seen before in different contexts. A larger, more diverse corpus increases the chance that the model has encountered analogous structures—similar argument forms, similar input-output formats, similar domain-specific conventions—during pretraining. When you then present a few examples of a new task in the prompt, the model can draw on those learned patterns to infer what comes next. The training objective is still next-token prediction, but a richer set of patterns in the training data gives the model more tools to compress the prompt, and that compression is what we observe as few-shot adaptation.
Go deeper
- Language Models are Few-Shot Learners · arXiv · Tom B. Brown et al. · 2020-05-28
- The Pile: An 800GB Dataset of Diverse Text for Language Modeling · arXiv · Leo Gao et al. · 2020-12-31
Image: Original diagram, The Daily Triptych. Licence: Original work. Source.