II · THE IDEA · ARTIFICIAL INTELLIGENCE
Why Scale Worked
▶ Listen · narrated
The models that reshaped the field did not arrive from a new algorithm. They arrived from running the old algorithm longer, on more text, with more weights.
At a glance
- What it is
- Measured relationships showing how loss decreases as model size, dataset size, or training compute increases
- Form
- Power laws: loss falls as a smooth function of scale, across orders of magnitude
- Key finding
- Performance improved predictably when parameters and data grew together, not from architecture changes alone
- Revision
- Hoffmann et al. found earlier models were undertrained; optimal compute splits more evenly between size and data
Imagine you are baking larger and larger cakes, and you notice that doubling the ingredients makes the cake exactly fifteen per cent taller every time, no matter whether you started with a small cake or a large one. That consistency would let you predict the height of a cake you have not yet baked, just by knowing the ingredient amounts. The scaling laws were like that: researchers trained models of many different sizes and found that performance improved smoothly as size grew, following a mathematical curve called a power law. More parameters made the model better at predicting the next word, more training data made it better, and more compute made it better. The improvements were predictable enough that you could estimate how well a much larger model would perform before spending the money to train it. That predictability made it feasible to build the very large models that now power most language applications, because the risk of wasting resources on a model that did not improve had fallen.
The scaling laws describe power-law relationships between model performance, measured as cross-entropy loss on held-out data, and three quantities: parameter count N, dataset size D measured in tokens, and total training compute C measured in floating-point operations. Kaplan et al. found that loss scaled as L(N) ∝ N^(-α), L(D) ∝ D^(-β), and L(C) ∝ C^(-γ), with each relationship holding across roughly four orders of magnitude when the other factors were not bottlenecks. The exponents differed: parameters mattered most, then data, then compute. The paper also derived a compute-optimal scaling strategy, concluding that for a fixed compute budget, you should scale parameters faster than data — roughly N ∝ C^0.73 and D ∝ C^0.27. Hoffmann et al. revisited this in 2022 with more training runs and found the original guidance undertrained models. Their revised estimate suggested N and D should scale roughly equally with compute, leading to smaller models trained on more tokens for the same budget. Both papers fitted power laws to empirical measurements; neither claimed the relationships were fundamental. The exponents are dataset-dependent, architecture-dependent, and may not extrapolate beyond the tested range. The laws also measure pretraining loss, which correlates with but does not fully determine downstream task performance, especially after fine-tuning or reinforcement learning from human feedback.
Look closer
The curves held across four orders of magnitude
Kaplan and colleagues trained models ranging from roughly a million parameters to ten billion, on datasets from twenty-two million tokens to twenty-three billion. Loss fell as a power law in three dimensions: more parameters helped, more data helped, more compute helped. The exponents were different — parameters mattered most — but the functional form stayed smooth. That consistency let them forecast performance at scales they had not yet trained, and the forecasts held when tested.
The original guidance was later revised
The 2020 paper suggested that once you fixed a compute budget, you should spend most of it on parameters and relatively little on data: a seventy-billion-parameter model trained on three hundred billion tokens. Two years later, Hoffmann's team re-examined the question with more careful experiments and found the optimum was closer to equal scaling: a model half that size, trained on twice as many tokens, performed better for the same cost. The revised paper called this compute-optimal training, and it motivated models like Llama.
The laws describe loss, not downstream usefulness
The scaling relationships measure cross-entropy loss on held-out text: how surprised the model is by the next token. That number fell reliably, but loss is a proxy. Whether a ten-per-cent reduction in perplexity translates into noticeably better summaries, more accurate reasoning, or fewer refusals depends on the task, the prompt, and how the model was aligned after pretraining. The curves gave confidence that scale would help; they did not promise which capabilities would emerge or when.
The story
In 2020, a team at OpenAI published a paper with an unglamorous title and a striking claim. They had trained more than fifty language models, varying size and data and compute budget systematically, and found that performance followed power laws. Make the model twice as large and loss fell by a predictable amount. Double the dataset and loss fell again, along a different curve with its own exponent. The relationships were smooth, and they held across four orders of magnitude.
This was not a theorem. It was a measurement, and it came with error bars. But it suggested something practically useful: that you could forecast how well a much larger model would perform without actually training it. The curves also implied that the returns to scale had not flattened. Loss kept falling as models grew, and there was no visible ceiling in the range they had tested.
The paper offered guidance for spending a fixed compute budget. If you could afford a certain number of floating-point operations, how should you divide them between parameters and training tokens? The answer they extracted from the data was lopsided: spend most of the budget on parameters, and train on a relatively modest amount of data. A seventy-billion-parameter model trained on three hundred billion tokens, for instance, rather than a smaller model trained longer.
That guidance shaped the next generation of large models. But in 2022, a team at DeepMind revisited the question with more training runs and tighter experimental control. They found the original paper had underweighted data. The compute-optimal split, they argued, was closer to balanced: for a given budget, you should scale parameters and training tokens at roughly the same rate. A thirty-billion-parameter model trained on six hundred billion tokens would outperform the seventy-billion-parameter model trained on less, for the same cost.
The revision mattered. It meant many existing models were undertrained, and that training smaller models longer could be more efficient than the field had assumed. Llama and several other influential models followed the revised guidance.
The scaling laws themselves remain empirical. They describe what happened in a particular range, on particular architectures, with particular data. They are not physical constants. The exponents could change as models grow further, or as architectures evolve, or as the supply of high-quality training data runs thin. What the laws demonstrated was that scale had been underexploited, and that performance gains were available without algorithmic breakthroughs — just by running the existing methods longer and larger.
Why it mattered then
The scaling laws changed the economics and strategy of model development. Before them, improving a language model usually meant trying a new architecture, a new training trick, or a new objective. Progress was fitful. The 2020 paper suggested that progress could be steadier: if you had the compute budget, you could buy performance by scaling up, and you could estimate the return in advance. That predictability made it easier to justify the cost of training ever-larger models, because the risk of a failed experiment fell. It also concentrated effort on the organisations that could afford the compute, since the path to better models now required resources more than ideas. The revised guidance two years later did not overturn the basic claim — it refined the recipe, making scale more accessible by showing that smaller models trained longer could compete with giants trained briefly.
Why it matters now
The scaling laws remain the operating assumption behind frontier model development, even as the exponents and optimal ratios get revisited. Every major lab still publishes scaling curves when introducing a new model family, and those curves still guide decisions about how to allocate compute. But the laws are now visibly approaching their limits. High-quality text data is no longer unlimited, and some researchers argue that the easy gains from scale are behind us. The curves may continue, but the cost per unit of improvement is rising, and the smoothness that made scaling predictable may not survive the transition to multimodal models, longer context windows, or post-training techniques that matter as much as pretraining scale. The laws worked, but they described a particular regime, not a permanent feature of the technology.
The surprising detail
The original scaling laws paper included a striking claim that was less widely remembered than the power-law curves: that architectural details mattered far less than scale. The authors trained models with different depths, widths, attention head counts, and feed-forward dimensions, and found that these choices changed performance by small amounts compared to simply adding more parameters. A mediocre architecture at large scale outperformed a carefully tuned architecture at small scale. That finding was philosophically uncomfortable — it suggested that human ingenuity in design was less important than brute resources — and it has not entirely held up. Sparse models, mixture-of-experts, and architectural changes that improve inference efficiency have all proven valuable. But for several years, the claim was defensible, and it shaped the field's priorities.
What is disputed
The exponents in the scaling laws were measured on transformer language models trained in a particular way, on particular datasets, in a particular range of sizes. Whether they generalise to multimodal models, to architectures with very long context, or to scales beyond what has been tested is uncertain. Some researchers argue the curves are already flattening as data quality becomes the bottleneck. Others expect them to continue if synthetic data or new modalities can substitute for web text. The laws describe the past reliably; their future is disputed.
Remember this
Scaling laws are measurements of what happened in a range, not predictions of what must happen next. They worked long enough to reshape the field.
Test yourself
The revised scaling guidance said to train smaller models on more data. Name one practical advantage of that approach beyond raw performance, and one risk it introduces.
Advantage: a smaller model is cheaper and faster to run after training, so if it performs as well as a larger undertrained model, you save on inference costs for the entire lifetime of deployment. The total cost of ownership falls even if training took longer. Risk: the revised guidance assumes you have enough high-quality data to train on. If your dataset is limited or repetitive, training longer may mean repeating the same examples many times, which can degrade performance through overfitting or make the model reproduce training data verbatim. The original guidance, which favoured larger models trained briefly, was less vulnerable to running out of fresh data.
Go deeper
- Scaling Laws for Neural Language Models · arXiv · Jared Kaplan et al. · 2020-01-23
- Training Compute-Optimal Large Language Models · arXiv · Jordan Hoffmann et al. · 2022-03-29
Image: Original diagram, The Daily Triptych. Licence: Original work. Source.