The Bitter Lesson of Compute and Scale

The bitter lesson, in the sense this lesson needs, is not a moral about bitterness. It is a historical pattern: methods that lean on general learning and ever-cheaper computation have repeatedly outrun methods that encode elaborate human knowledge of the task. In language modelling, that pattern became measurable rather than merely anecdotal when researchers began to treat scale as an object of study in its own right.

Kaplan and colleagues trained families of Transformer language models and asked how cross-entropy loss moved when three quantities changed—model size (non-embedding parameters), dataset size, and the compute used for training. The answers were strikingly regular. Loss improved as power laws in each of those quantities over wide dynamic ranges. The trends did not require a particular clever trick at each new size; they looked like smooth extrapolations. Larger models were also markedly more sample-efficient: they extracted more signal from each token of data than smaller ones did.

Equally important was what mattered less. Over a substantial range, details such as the balance of depth and width did not dominate outcomes the way raw scale did. That finding undercut a familiar research instinct—the hope that a carefully engineered inductive bias would substitute for orders of magnitude in compute. Some bias is still necessary; a model family, an optimiser and a tokenisation scheme are all human choices. But the scaling results suggested that, once a workable general method existed, further gains often came from riding the power laws rather than from stacking on more hand-designed structure.

Those same curves carried an economic reading. If loss is a smooth function of compute, then budgets, not only ideas, set the frontier. Optimal allocation under a compute budget, in their analysis, favoured training quite large models on relatively modest data and stopping short of full convergence on that data—again a prescription about how to spend FLOPs, not about how to encode linguistic theory.

Strubell and colleagues made the other half of the ledger visible. Training modern NLP models consumes substantial energy; when that energy is converted into estimated carbon emissions under stated assumptions about hardware and grids, even routine research practice looks costly. Their estimates placed a single BERT-base training run without hyperparameter tuning in the carbon neighbourhood of a passenger flight across the United States, and placed aggressive neural architecture search far higher—into ranges they compared with multiple long-haul flights or a substantial fraction of a car’s lifetime emissions, depending on the scenario. They argued for reporting energy and carbon alongside accuracy, and for noticing that access to the compute that scaling rewards is unevenly distributed.

Read together, the two lines of work frame a single historical claim. General, compute-heavy learning methods won on the metric the field optimised—held-out loss and the tasks that track it—while the cost of that win shifted onto electricity, carbon and concentration of resources. The lesson is bitter twice over: once for the researchers whose carefully specialised algorithms were overtaken by scale, and once for anyone who assumed that “more compute” was only an engineering detail rather than a policy object.