II · THE IDEA · ARTIFICIAL INTELLIGENCE
Test-Time Compute
▶ Listen · narrated
A model that pauses to try several approaches before answering may outperform one twice its size that replies immediately. The compute you spend matters as much as where you spend it.
At a glance
- What it is
- Allocating compute during inference rather than training, typically through search or verification
- Common methods
- Generating multiple solutions and selecting the best; verifying reasoning steps; tree search
- Trade-off
- Higher latency and cost per query, but better performance without retraining
- Scaling behaviour
- Can be more efficient than parameter scaling in compute-limited regimes
Imagine you are taking a difficult exam. One approach is to study harder beforehand — that is like training a larger model. Another approach is to spend more time during the exam itself: write out multiple attempts, check your working, cross out wrong answers and try again. That is test-time compute. For some problems, the second approach gets you a better result even if you studied less, because the extra time during the test lets you catch mistakes and explore different strategies. Language models can do the same thing. Instead of answering immediately, they can generate several possible answers, verify each step of their reasoning, or search through different approaches before committing to a response. It costs more time and energy per question, but the answers can be more reliable than those from a larger model that only tries once.
Test-time compute refers to inference-time algorithms that trade latency and computational cost for improved output quality, without retraining the model. The simplest form is best-of-N sampling: generate N independent completions, score them with a reward model, and return the highest-scoring candidate. More sophisticated methods include sequential revision, where the model critiques and improves its own output over multiple rounds, and tree search, where the model expands a graph of partial solutions, using a value function or process reward model to guide exploration and pruning.
Process reward models, trained to score intermediate reasoning steps rather than final outcomes, have been shown to generalise better and to be more robust against spurious correlations. Lightman et al. demonstrated this on mathematical reasoning tasks, where step-level supervision outperformed outcome supervision even when both were trained on the same underlying problems.
Snell et al. formalised the trade-off between test-time compute and parameter scaling, showing that under certain conditions — particularly when inference budget is large relative to training budget — allocating compute to inference-time search can yield better performance per unit of total compute than training a larger model. The optimal allocation depends on the task, the model's capability distribution, and the specific test-time algorithm used.
The architecture of the base model matters. Models trained with techniques that expose intermediate reasoning — chain-of-thought prompting during training, for instance — tend to be more amenable to test-time search, because their outputs have more structure for a reward model to evaluate. Conversely, a model trained to emit only final answers provides less surface area for process supervision.
Latency is the primary cost. Generating and scoring N candidates multiplies inference time by roughly N. Tree search can be worse, depending on branching factor and depth. For applications where correctness justifies the wait — formal verification, medical reasoning, high-stakes planning — this is often acceptable. For low-latency applications it is not, which is why test-time compute is not a universal replacement for parameter scaling but a complementary axis.
Look closer
Two ways to verify: outcome or process
You can generate many candidate solutions and score them based on their final answer — outcome supervision — or you can check each reasoning step along the way — process supervision. Lightman and colleagues trained reward models both ways and found process supervision more reliable: it catches errors earlier, generalises better to problems outside the training distribution, and makes it harder for a model to arrive at a correct answer through invalid reasoning. The distinction matters because it changes what you need from your training data: process supervision requires human labellers to judge intermediate steps, not just final results.
The compute can be spent in different places
Test-time compute is not a single technique. You might generate N independent solutions and pick the most common answer. You might build a search tree, expanding promising branches and pruning weak ones. You might use one model to propose steps and another to verify them. You might iterate: generate, check, revise, repeat. Each approach has different latency characteristics, different failure modes, and different requirements for what the model must have learned during training. What they share is the willingness to spend more time and energy per query in exchange for better output.
The scaling curves cross
Snell and colleagues compared two paths to better performance: training a larger model, or running a smaller model with more test-time compute. In some regimes, particularly when inference budget is large relative to training budget, test-time scaling delivered better results per unit of total compute. The curves are not universally in favour of one approach — it depends on how much compute you have, how it is distributed between training and inference, and what performance level you need. But the existence of a crossover point means test-time compute is not merely a fallback when you cannot train something bigger; it can be the more efficient choice.
The story
For years the primary way to improve a language model was to make it larger. More parameters, more training data, more compute during the training run. Inference was treated as a separate concern: once the model was trained, you ran it as efficiently as possible, generating one token at a time until the answer was complete.
That framing is no longer the only option. Test-time compute refers to spending additional computation during inference — when the model is answering a query — in order to improve the quality of the response. Instead of generating a single answer in one forward pass, the model might generate several candidates and select among them. It might verify its own reasoning steps. It might search through a space of possible solutions, exploring and backtracking like a human working through a difficult problem on paper.
The simplest form is majority voting: generate N independent answers and return the most common one. This works when errors are random rather than systematic. A more sophisticated approach is best-of-N sampling with a learned reward model: generate multiple candidates, score each one, and return the highest-scoring response. The reward model can be trained to recognise correct final answers — outcome supervision — or to verify each step of the reasoning process — process supervision.
Lightman and colleagues compared these two forms of supervision directly. They trained reward models on mathematical problem-solving, giving one model access only to whether the final answer was correct, and giving the other model human judgements on every intermediate step. Process supervision proved more reliable. It generalised better to new problems, and it was harder to fool: a model supervised only on outcomes could sometimes reach the right answer through invalid reasoning, but step-by-step verification caught those errors.
Process supervision requires more expensive training data, because human labellers must judge many steps per problem rather than one final answer. But the resulting reward model can guide search more effectively. Instead of generating complete solutions in parallel, the model can build a search tree: propose a first step, evaluate it, expand the promising branches, prune the weak ones, and continue until a solution emerges. This is closer to how humans solve hard problems, and it allows the model to allocate compute where it is most needed.
Snell and colleagues formalised the trade-off between test-time compute and model scale. They showed that in some regimes — particularly when you have a fixed total compute budget and can choose how to divide it between training and inference — spending more at test time can be more efficient than training a larger model. The crossover point depends on the task, the model architecture, and how the test-time compute is used, but its existence changes the economics of model deployment. A smaller model running with verification and search may cost less to train and serve than a larger model running once per query.
This is the foundation on which reasoning models are built. They are not simply larger or trained on better data; they are designed to use test-time compute effectively, trading latency for reliability on tasks where getting the answer right matters more than answering quickly.
Why it mattered then
The recognition that test-time compute could be a primary scaling axis, rather than a fallback, emerged as training runs became expensive enough that alternatives were worth exploring seriously. Snell's work in 2024 provided empirical evidence that the trade-off was not always in favour of larger models, and Lightman's earlier work on process supervision gave a concrete method for training the reward models that test-time search requires. Together they suggested that the next generation of models might improve as much through better inference-time algorithms as through additional parameters.
Why it matters now
Reasoning models — systems that visibly pause, generate internal working, and verify their steps — are direct applications of test-time compute. They are commercially deployed, not research prototypes. The shift matters because it changes what improvement looks like: instead of waiting for the next training run, you can improve a deployed model by changing how it searches at inference time. It also changes the cost structure. Test-time compute is expensive per query, but it is paid only when used, and it can be scaled independently of the base model. For tasks where accuracy justifies the cost — medical reasoning, formal verification, complex planning — this is often the correct trade-off.
The surprising detail
The fact that process supervision outperforms outcome supervision is not obvious in advance. You might expect that as long as you reach the correct answer, the path does not matter, and that training on final results would be simpler and just as effective. But Lightman and colleagues found the opposite: models trained with step-by-step verification generalised better and were more robust. The likely reason is that outcome supervision allows the model to exploit spurious correlations — patterns that happen to lead to correct answers in the training set but do not reflect valid reasoning. Process supervision forces the model to learn the structure of correct reasoning itself, which transfers more reliably to new problems.
Remember this
Test-time compute is not an auxiliary technique. It is a scaling axis, and on some tasks it scales more efficiently than parameters.
Test yourself
You have a fixed budget of compute. You can either train a model twice as large, or train the current model and spend the saved compute on test-time search during inference. Name two factors that would push you toward the second option.
First, if the task benefits from explicit search or verification — mathematics, code generation, planning — then test-time compute can exploit that structure, whereas a larger model without search is still guessing in one pass. Second, if your inference volume is low relative to training cost, you can afford to spend heavily per query without the inference bill exceeding what you saved by training a smaller model. A third factor: if you need to improve a deployed model quickly, changing the inference algorithm is faster than a new training run. And a fourth: if the task requires reliability over speed, users may accept higher latency in exchange for verified correctness, which test-time compute can provide but parameter scaling alone cannot guarantee.
Go deeper
- Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters · arXiv · Charlie Snell et al. · 2024-08-06
- Let's Verify Step by Step · arXiv · Hunter Lightman et al. · 2023-05-31
Image: Original diagram, The Daily Triptych. Licence: Original work. Source.