II · THE IDEA · ARTIFICIAL INTELLIGENCE
The Data Is the Model
▶ Listen · narrated
When a hiring algorithm systematically downranked women, the problem was not in the code. It was in ten years of historical decisions the system had been asked to learn from.
At a glance
- Core principle
- A model encodes only the patterns present in its training set
- The Pile
- 800GB dataset assembled by Leo Gao and colleagues, 2020
- Dataset documentation
- Datasheets for Datasets framework proposed by Timnit Gebru et al.
- Common failure
- Confident predictions that reflect training bias rather than ground truth
Imagine you are learning to cook by watching someone else prepare meals for a year. If they make pasta three times a week and curry once a month, you will learn to make pasta well and curry poorly, because you have seen far more examples of one than the other. You might believe pasta is simply more common, or easier, because that is what your experience taught you. A language model learns the same way. It sees millions of sentences during training and adjusts its internal settings so that sentences similar to the training examples score higher. If the training data contains more text about men in leadership roles, the model learns that pattern. If medical records overrepresent certain populations, the model becomes more accurate for those populations and less accurate for others. The model has no way to know whether the training data was representative or skewed. It only knows what it was shown, and it reproduces those patterns with confidence.
A model's parameters are learned by minimising prediction error over the training distribution. The training objective — typically next-token prediction for a language model — rewards the model for assigning high probability to sequences that resemble the training data and low probability to sequences that do not. If a particular pattern appears frequently in the training set, gradient descent will adjust the parameters to encode that pattern, because doing so reduces loss. If a pattern is absent or rare, the model will have weak or noisy representations for it, because the training signal was insufficient.
This means that every statistical regularity in the training data — including biases, skewed demographics, and historical inequities — becomes encoded in the learned parameters. A model trained on biased hiring data will learn to predict biased hiring decisions, because that is what minimises loss on that dataset. Techniques like prompt engineering, few-shot learning, or careful system instructions operate at inference time and can shift the model's behaviour at the margins, but they cannot override the learned associations embedded in the parameters. If the training data contains ten times as many examples of Disease A as Disease B, the model's internal representations will reflect that imbalance: its confidence will be miscalibrated, its attention patterns will favour Disease A features, and its likelihood estimates will be biased toward the more frequent class.
Data augmentation, reweighting, or training on a more balanced dataset are interventions that must happen before or during training. Post-training methods like RLHF can adjust behaviour for specific prompts or tasks, but they are corrections applied on top of a foundation that was learned from the original data distribution. The training data is not an input in the usual sense. It is the source of truth from which the model's entire world model is derived.
Look closer
The hiring algorithm that learned from history
Amazon built a résumé screening tool trained on a decade of historical hiring decisions. Because most past hires in technical roles had been men, the model learned to penalise résumés that included the word "women's" — as in "women's chess club" — and downranked graduates of two all-women's colleges. The system was not programmed to discriminate. It learned discrimination because discrimination was the statistical regularity in the training set. The project was eventually abandoned.
The Pile was assembled to be documented
The Pile is an 800-gigabyte text dataset built by Leo Gao and colleagues, combining twenty-two smaller sources including books, academic papers, code repositories and web text. Unlike many earlier datasets, it ships with detailed provenance: what each subset contains, where it came from, what biases the authors believe it carries. This transparency does not eliminate the biases — the text still reflects the demographics and ideologies of its sources — but it allows researchers to know what they are training on, which is a prerequisite for interpreting what the resulting model has learned.
Datasheets make training data legible
Timnit Gebru and colleagues proposed the Datasheets for Datasets framework in 2018, arguing that every dataset used to train a model should come with structured documentation: who collected it, for what purpose, what population it represents, what known limitations it has. The analogy is to electronics datasheets, which specify voltage tolerances and operating temperatures. A dataset without a datasheet is a component without a spec, and a model trained on it inherits undocumented assumptions that will eventually surface as unexplained failures in deployment.
The story
A language model has no independent access to the world. It cannot look things up, run experiments, or check its answers against reality. Everything it knows arrived through the training data, and the only patterns it can reproduce are the patterns that were statistically present in that data. If the training set contains more sentences about men in leadership than women, the model will learn that association. If medical text overrepresents certain populations, diagnoses for others will be less accurate. If historical decisions were biased, a system trained to predict them will inherit the bias.
This is not a defect in the learning algorithm. It is the learning algorithm working exactly as designed. A model's task is to compress the training distribution into a set of parameters, then use those parameters to generate new examples that resemble the training set. When we say a model is "trained on Wikipedia", we mean it has seen millions of Wikipedia sentences and adjusted its internal numbers so that Wikipedia-like sentences score higher than non-Wikipedia-like sentences. The model has no notion of truth, only of resemblance.
Amazon discovered this in 2018 when it abandoned an internal project to automate résumé screening. The system had been trained on a decade of historical hiring decisions — specifically, on which résumés had led to interviews and which had not. The training data reflected the company's actual hiring patterns, which in technical roles had favoured men. The model learned that pattern as faithfully as it learned any other. It began penalising résumés that contained the word "women's", downranking graduates of two all-women's colleges, and systematically scoring female candidates lower. Engineers tried to patch the behaviour by removing explicitly gendered features, but the bias was encoded in subtler correlations throughout the data. The system could not be fixed because the system was working correctly. It was the ground truth that was skewed.
The Pile, an 800-gigabyte dataset assembled by Leo Gao and colleagues in 2020, represents a different approach. Rather than treating training data as an invisible substrate, the authors documented it extensively. The Pile combines twenty-two sources — books, academic papers, GitHub repositories, web crawls, legal text, medical abstracts — and ships with detailed descriptions of what each subset contains, where it came from, and what demographic or topical biases the authors believe it carries. This does not make the data unbiased. The text still overrepresents English, certain academic disciplines, and populations with internet access. But it makes those biases knowable, which allows researchers to interpret a trained model's behaviour in light of what it was actually shown.
Timnit Gebru and colleagues formalised this idea in their 2018 proposal for Datasheets for Datasets. The framework asks dataset creators to answer structured questions: Who collected this data, and why? What population does it represent? What is missing? The analogy is to electronics components, which ship with datasheets specifying voltage tolerances and operating temperatures. A resistor without a datasheet is unusable in any system that requires reliability. A dataset without documentation is the same: a component with unknown properties, whose failures will be unexplained until someone reverse-engineers what it actually contained.
The implication is uncomfortable. A model cannot learn what it was not shown, and it cannot unlearn the regularities present in its training set through prompting alone. If you train on ten years of biased hiring decisions, the model will encode ten years of biased hiring decisions. If you train on medical literature that underrepresents a population, the model will be less accurate for that population. The training data is not input to the model. It is the model, compressed.
Why it mattered then
The Amazon hiring tool was abandoned in 2018, the same year Gebru and colleagues published the Datasheets proposal. Both events arrived at a moment when machine learning was moving out of research labs and into high-stakes deployment — hiring, lending, criminal sentencing, medical diagnosis — and the assumptions baked into training data were suddenly producing visible harm. Earlier systems had been small enough, or deployed narrowly enough, that biased data produced biased outputs in controlled settings where humans could catch them. But as models grew larger and were applied more broadly, the statistical regularities in the training set began to govern decisions at scale, and the biases became structural rather than anecdotal. The Pile and the Datasheets framework were both attempts to make training data legible before it was compressed into parameters, because once the model is trained, the data's fingerprint is permanent.
Why it matters now
Training data has only become more consequential. The largest language models are now trained on trillions of tokens scraped from the public internet, and their behaviour — including what they know, what they assume, and whose perspectives they centre — is determined almost entirely by the composition of that data. Researchers have documented that models perform worse on dialects underrepresented in training data, that they reproduce occupational stereotypes present in web text, and that they hallucinate less for topics that appeared frequently in their training sets. These are not implementation bugs. They are the system working as designed, which means the only point of intervention is the data itself. Datasheets remain rare. Most training sets are assembled opportunistically from whatever large text corpora are available, documented lightly if at all, and treated as interchangeable. They are not interchangeable. Two models with identical architectures, trained on different data, are different models. The data is the model.
The surprising detail
The Amazon hiring algorithm was not trained to discriminate, and it was not programmed with any rule about gender. It simply learned the statistical pattern that had been present in a decade of hiring data: that résumés associated with women were less likely to lead to interviews. When engineers tried to fix the system by removing explicitly gendered words, the bias persisted, because it was encoded in hundreds of subtler correlations the model had learned — writing style, university names, verb choices, extracurricular activities. The system could not be debugged because it was not malfunctioning. It had compressed the training distribution accurately. The problem was that the training distribution itself encoded a decade of biased decisions, and there was no way to train the model on that history without the model learning that history.
Remember this
A model encodes the regularities in its training set. If the data is biased, the model will be biased. There is no other source of truth.
Test yourself
You are training a medical diagnosis model. Your training set contains ten times as many case notes for Disease A as for Disease B, even though both diseases occur at similar rates in the population. Describe two distinct ways this imbalance will likely affect the trained model's behaviour, and explain why prompt engineering alone cannot fix either one.
First, the model will be more confident and more accurate when diagnosing Disease A, because it has seen far more examples of how that disease presents, what symptoms co-occur, and what language clinicians use to describe it. For Disease B, it has weaker statistical evidence and may confuse it with other conditions or miss subtle presentations. Second, when symptoms are ambiguous, the model will be biased toward predicting Disease A, because that prediction resembles more of its training data. Prompt engineering cannot fix these problems because the model's internal representations were learned from the imbalanced data. You can instruct it to "consider Disease B carefully", but its learned associations, its confidence calibration, and its internal similarity metrics all reflect the ten-to-one ratio it was trained on. The imbalance is in the parameters, not in the prompt.
Go deeper
- The Pile: An 800GB Dataset of Diverse Text for Language Modeling · arXiv · Leo Gao et al. · 2020-12-31
- Datasheets for Datasets · arXiv · Timnit Gebru et al. · 2018-03-23
Image: Original diagram, The Daily Triptych. Licence: Original work. Source.