II · THE IDEA · ARTIFICIAL INTELLIGENCE
Chinchilla and Compute-Optimal Training
▶ Listen · narrated
The largest models were not the best models. They had been fed too few tokens, and the compute budget had been spent in the wrong place.
At a glance
- What it showed
- For a fixed compute budget, smaller models trained on more data outperform larger models trained on less
- Chinchilla itself
- 70 billion parameters, trained on 1.4 trillion tokens
- Comparison point
- Gopher: 280 billion parameters, same compute, but only 300 billion tokens
- Result
- Chinchilla outperformed Gopher and most larger models despite being four times smaller
Imagine you are baking a cake and you have a fixed budget for ingredients. You could buy a very large cake tin and fill it halfway, or you could buy a smaller tin and fill it properly. Chinchilla showed that language models were like half-filled tins: the field had been making them too large and not feeding them enough data. A 70-billion-parameter model trained on 1.4 trillion tokens outperformed a 280-billion-parameter model trained on only 300 billion tokens, even though both cost the same to train. The lesson was that parameters and data should grow together. If you double the model size, you should double the amount of text it sees during training. Before 2022, most models were trained on far too little data for their size, which meant they were both more expensive to run and less capable than they could have been.
Chinchilla scaling laws describe the optimal allocation of a fixed compute budget between model parameters and training tokens. Hoffmann et al. trained over 400 models ranging from 70 million to over 16 billion parameters, each on token counts between 5 billion and 500 billion, then fit a power law to the resulting loss surface. The key finding: for a given compute budget C, the optimal model size N and training token count D scale approximately as N ∝ C^0.5 and D ∝ C^0.5, implying they should grow equally. The rough optimum is around 20 tokens per parameter, though later analysis suggests this may underestimate the data requirement.
This contradicted the Kaplan et al. 2020 scaling laws, which had recommended scaling parameters faster than tokens because their experiments had not varied dataset size proportionally. Kaplan's models were systematically undertrained by Chinchilla's metric. The practical consequence: Chinchilla, at 70B parameters and 1.4T tokens, outperformed Gopher (280B parameters, 300B tokens) and matched or exceeded models twice its size, despite identical training compute.
The implication for deployment is significant. Inference cost scales with parameter count, so a compute-optimal model is both better and cheaper to serve than an undertrained larger model. This shifted the bottleneck from model capacity to data curation, making high-quality datasets the limiting resource for frontier training. It also invalidated the parameter-count-as-proxy-for-capability heuristic: a 70B model trained optimally outperforms a 175B model undertrained by a factor of ten.
Look closer
The earlier scaling laws were not wrong, just incomplete
Kaplan et al. in 2020 had shown that loss decreased predictably as model size grew, and their curves were reproducible. What they had not varied systematically was the amount of training data relative to parameter count. Their experiments held the token budget relatively fixed while scaling parameters, so the implicit recommendation was to grow the model and reuse roughly the same corpus. Hoffmann's team swept both axes and found that the 2020 laws had been measuring a slice through a larger space, and the slice was not the optimal one.
The ratio is approximate, not a law of nature
Chinchilla's headline finding was that parameters and tokens should scale equally: if you double the parameters, double the training tokens. The rough optimum they reported was around 20 tokens per parameter, but that figure depends on how you count compute, what you consider optimal, and the specific loss function. Later work has suggested the ratio may be somewhat higher. What remains uncontested is the broader claim: before 2022, the field was systematically underfeeding its models.
Smaller models became commercially viable
If a 70-billion-parameter model trained on enough data outperforms a 280-billion-parameter model trained on less, then inference cost drops by a factor of four while quality improves. This was not an academic curiosity. It meant that a well-trained smaller model could be served faster, on cheaper hardware, with lower latency, and still beat the undertrained giant. The result made open-weight models more practical and shifted the bottleneck from parameter count to data curation.
The story
In 2020, researchers at OpenAI published scaling laws showing that language model loss decreased as a power law in model size, and that this relationship was smooth and predictable across several orders of magnitude. The implication, widely adopted, was that the path to better models lay in making them larger. The number of parameters became the headline figure, and training data was treated as something you gathered once and reused.
The 2020 experiments had varied model size while holding the dataset size relatively constant, or at least not scaling it in proportion to parameters. For a 10-billion-parameter model and a 100-billion-parameter model, the training token count might differ by a factor of two or three, not ten. This was partly a practical constraint—large curated datasets are expensive—and partly an inherited assumption from earlier work in computer vision, where you trained to convergence on a fixed dataset and then stopped.
Hoffmann and colleagues at DeepMind ran a more systematic sweep. They trained models of many different sizes, each on a different quantity of tokens, and measured performance per unit of compute rather than per parameter. The compute budget was the constraining resource: a fixed amount of floating-point operations, which could be spent either on a large model trained briefly or a small model trained for longer.
The result was a surface, not a line. For any given compute budget, there was an optimal allocation between model size and training duration, and that optimum was not where the field had been operating. The 2020 scaling laws had implicitly recommended spending most of the budget on parameters. The 2022 laws said to split it evenly: if you double the compute, increase parameters by roughly 1.4× and tokens by roughly 1.4×, not parameters by 2× and tokens hardly at all.
Chinchilla was the proof. It had 70 billion parameters and was trained on 1.4 trillion tokens, maintaining roughly a 20:1 token-to-parameter ratio. Gopher, trained earlier by the same lab, had 280 billion parameters but only 300 billion tokens—a ratio closer to 1:1. Both models used approximately the same compute budget. Chinchilla was smaller, but it outperformed Gopher on almost every benchmark, and it also outperformed models with more parameters from other labs.
The implication was that GPT-3, at 175 billion parameters trained on 300 billion tokens, had been undertrained by a factor of ten or more. So had most of the headline models from 2020 and 2021. They were capable of impressive performance, but they would have been measurably better—at the same cost—if the compute had been reallocated toward data rather than parameters.
Why it mattered then
The result arrived at a moment when the race to scale was accelerating and the capital cost of training a frontier model was becoming a meaningful business constraint. Google had recently announced a 540-billion-parameter model. The assumption was that the next threshold would be a trillion parameters, and the compute budget required was being measured in tens of millions of dollars. Chinchilla suggested that this was the wrong path. A better model could be trained for the same cost by making it smaller and feeding it more data. That was not just an efficiency gain; it was a strategic redirection. Smaller models are faster at inference, cheaper to serve, and easier to fine-tune. If the field had continued on the parameter-scaling trajectory, the largest models would have become accessible only to a handful of institutions. Compute-optimal training made state-of-the-art quality achievable at a fraction of the size, which lowered the barrier to entry and made open release more practical.
Why it matters now
The ratio has become a design assumption. When a lab announces a new model, the parameter count and training token count are now reported together, and the token count is often the larger number. Llama 2, at 70 billion parameters, was trained on 2 trillion tokens. Llama 3 pushed the ratio further. The bottleneck is no longer whether you can build a large enough model, but whether you can source, filter, and deduplicate enough high-quality data to train it properly. This has shifted the competitive landscape. Data curation is now as important as architecture, and the methods for cleaning web scrapes, filtering out duplication, and balancing domain coverage are treated as proprietary. It has also made the undertrained models from 2020 and 2021 look like artifacts of a different era, even though some are still in production. A 175-billion-parameter model trained on 300 billion tokens is not a large model by current standards; it is a medium-sized model that was stopped early. The principle also affects fine-tuning and continued pretraining. If you take a checkpoint that was trained compute-optimally and continue training it on domain-specific data, you are moving away from the optimum unless you accept that the model is now effectively larger, in compute terms, than its parameter count suggests. Some practitioners now retrain from scratch on a smaller architecture with more in-domain data rather than fine-tuning a large general model, because the compute-optimal logic applies at every stage.
The surprising detail
The Chinchilla paper included a set of models deliberately trained away from the optimum to demonstrate the performance loss, and the undertrained models degraded more sharply than the overtrained ones. A model trained on too few tokens lost capability faster than a model trained on too many, which suggests that undertraining leaves something structurally unfinished in a way that overtraining does not. The asymmetry is not fully explained, but it implies that if you must err, it is safer to err toward more data.
What is disputed
The exact token-to-parameter ratio depends on how compute is counted, the loss function used, and the downstream task. Hoffmann et al. reported roughly 20:1 as optimal, but subsequent work has suggested the ratio may be higher, perhaps 30:1 or more, especially when accounting for inference costs over the model's lifetime rather than training cost alone. What is not in dispute is the directional claim: models trained before 2022 used far too few tokens for their size.
Remember this
Most large models before 2022 were undertrained. The optimal ratio is roughly equal scaling: double the parameters, double the tokens.
Test yourself
A lab has a fixed compute budget and is deciding between training a 30-billion-parameter model on 600 billion tokens or a 120-billion-parameter model on 150 billion tokens. Both options use the same total compute. Based on Chinchilla scaling laws, which is likely to perform better, and what is one practical advantage the better choice also provides?
The 30-billion-parameter model trained on 600 billion tokens is likely to perform better, because it maintains a 20:1 token-to-parameter ratio, close to the compute-optimal allocation. The 120-billion-parameter model, at 1.25:1, is severely undertrained. The practical advantage is that the smaller model is also faster and cheaper to serve at inference time. You get both better quality and lower deployment cost from the same training budget.
Go deeper
- Training Compute-Optimal Large Language Models · arXiv · Jordan Hoffmann et al. · 2022-03-29
- Scaling Laws for Neural Language Models · arXiv · Jared Kaplan et al. · 2020-01-23
Image: Original diagram, The Daily Triptych. Licence: Original work. Source.