II · THE IDEA · ARTIFICIAL INTELLIGENCE
Sampling Many Answers
▶ Listen · narrated
One attempt at reasoning can stumble on a bad step. Generate ten paths instead, and the mistakes rarely line up—which means the correct answer usually wins the vote.
At a glance
- What it is
- Generate multiple reasoning chains for the same question, then choose the answer that appears most often
- Typical count
- 5 to 40 samples, depending on question difficulty and compute budget
- Cost multiplier
- Scales linearly with sample count—ten paths cost ten times as much as one
- Main benefit
- Improves accuracy on arithmetic, logic and multi-step reasoning without changing the model
Imagine you are trying to solve a difficult maths problem and you are not sure you got it right. You could solve it again from scratch, and if you make a mistake the second time, it will probably be a different mistake. Do it ten times, and the correct answer is likely to appear more often than any single wrong answer. Self-consistency applies this idea to language models: generate ten different reasoning paths for the same question, count which final answer appears most often, and return that one. It costs ten times as much as asking once, but it often produces a more reliable answer because the errors are unlikely to line up.
Self-consistency samples multiple reasoning chains from a language model at non-zero temperature, then selects the most frequent final answer by majority vote. The model is prompted to produce a chain of thought—a step-by-step explanation leading to an answer—and this prompt is run multiple times with temperature typically between 0.5 and 0.8 to ensure diversity. Each sample produces a complete reasoning path and a final answer; the paths themselves are discarded and only the final answers are tallied. The method assumes that errors in reasoning are not systematically correlated across samples: a model that makes an arithmetic error on step three in one sample will likely make a different error, or no error, in another sample. As long as the correct reasoning path has non-negligible probability mass in the model's output distribution, sampling repeatedly increases the chance that it appears at least once, and if errors scatter, the correct answer accumulates more votes than any single incorrect answer. The technique requires no training, no gradient computation, and no access to model internals; it is purely an inference-time intervention. Cost and latency scale linearly with the number of samples. Empirical results on arithmetic, commonsense reasoning, and symbolic reasoning benchmarks show accuracy improvements of 10 to 20 percentage points over single-sample chain-of-thought prompting, with diminishing returns beyond 20 to 40 samples depending on question difficulty. The method works best when the correct answer is well-defined and when the model's errors are due to stochastic variation rather than systematic bias. It fails when all samples converge on the same wrong answer, which occurs when the model has a strong prior toward a particular misconception or when the question is outside the model's capability regardless of sampling strategy.
Look closer
The vote is on final answers, not on reasoning steps
You prompt the model to work through the problem step by step, then sample ten completions with the temperature set above zero so each path differs. The model might reach "42" via three different chains of logic, "37" via two chains, and five other answers once each. The method returns 42, because it appeared most often. The reasoning paths themselves are discarded after counting. This means a correct answer reached through flawed logic still counts, and a perfectly sound argument for a wrong answer gets no special weight.
It works because errors are not correlated
A model making an arithmetic mistake on step three of a five-step problem will likely get a different step wrong in the next sample, or make no error at all. The correct reasoning path exists in the model's distribution of possible outputs; sampling repeatedly gives it multiple chances to appear. As long as mistakes scatter across samples rather than clustering on the same wrong answer, the correct answer accumulates votes faster. This assumption holds well for arithmetic and symbolic reasoning, less well for questions where a systematic bias pushes most samples toward the same plausible-sounding error.
Temperature must be high enough to produce variety
At temperature zero the model is deterministic, so all samples would be identical and voting would be pointless. Typical implementations use temperature between 0.5 and 0.8, enough to vary the reasoning path without producing incoherent output. Too high and the samples become unreliable; too low and you are paying for near-duplicates. The optimal temperature depends on the model and the question type, and is usually found by experiment rather than theory.
The story
The method appeared in a 2022 paper by Wang and colleagues at Google Research, under the name self-consistency. The premise was simple: chain-of-thought prompting had already shown that asking a model to reason step by step improved performance on complex questions, but a single reasoning path could still fail. What if you generated several paths and took a vote?
The results were striking. On a collection of arithmetic and commonsense reasoning benchmarks, self-consistency improved accuracy by 10 to 20 percentage points over standard chain-of-thought prompting, without any change to the model itself. The method worked across model scales and across problem types, from grade-school maths to logical puzzles. It was not a marginal gain; it was one of the largest improvements you could buy with inference-time compute alone.
The mechanism is not mysterious. A model trained on a large corpus has seen many ways to solve a problem, and its distribution over next tokens encodes that variety. A single sample draws one path from that distribution, and if it happens to hit a low-probability error early on, the rest of the reasoning follows that error forward. Sampling again gives the model another chance, and the error will likely occur at a different step or not at all. The correct path, being more probable in aggregate, appears more often than any single incorrect path.
The method requires no training, no fine-tuning, and no access to model weights. You can apply it to a model behind an API by simply calling the endpoint multiple times with temperature above zero. The cost is linear in the number of samples: ten paths cost ten times as much as one. For high-stakes questions where accuracy matters more than speed, that trade is often worth making.
A related technique called Tree of Thoughts, introduced by Yao and colleagues in 2023, extends the idea by treating reasoning as a search problem. Instead of generating complete paths independently, it generates partial reasoning steps, evaluates them, and expands only the most promising branches. This allows the model to backtrack from dead ends rather than committing to a full path before seeing where it leads. Tree of Thoughts is more complex to implement and more expensive to run, but it can solve problems that require explicit exploration of multiple strategies.
Why it mattered then
Self-consistency arrived at a moment when chain-of-thought prompting had demonstrated that models could reason if asked to show their work, but the reliability of that reasoning remained uneven. A single chain of thought might produce a correct answer on one run and a confident error on the next. For researchers trying to benchmark reasoning ability or deploy models in settings where accuracy mattered, this variability was a problem. Self-consistency offered a solution that required no new training and no architectural change—just more samples and a vote. It was immediately practical, and it revealed something about how reasoning capability is distributed across a model's output space rather than concentrated in a single deterministic path.
Why it matters now
The method remains widely used in production systems where the cost of a wrong answer exceeds the cost of additional inference. Legal research tools, medical decision support, and financial analysis applications often generate multiple reasoning chains and aggregate the results. The technique also underpins more sophisticated approaches: recent work on inference-time scaling, where models are given more compute at test time to improve performance, often relies on sampling multiple solutions and selecting the best. The 2024 release of models explicitly trained to benefit from extended reasoning at inference time has made the trade-off between compute and accuracy more explicit, and self-consistency is a straightforward way to spend that compute. It is also a reminder that a model's capabilities are not fully captured by a single greedy decode; the distribution matters.
The surprising detail
The method works better on some question types than others, and the pattern is revealing. On arithmetic problems with a single correct answer, self-consistency produces large gains. On open-ended questions where many answers are defensible, voting can suppress valid minority answers in favour of a bland consensus. On questions where the model has a systematic bias—such as preferring answers that sound confident or that match a common misconception in the training data—all the samples may converge on the same plausible error, and voting does nothing to fix it. The technique exposes where uncertainty in the model is statistical noise that averages out, and where it is a deeper confusion that persists across samples.
Remember this
Sampling many answers and taking a vote turns inference-time compute into accuracy, as long as the errors scatter and the correct answer is somewhere in the model's distribution.
Test yourself
You apply self-consistency to two different questions, both answered incorrectly by a single sample. On the first, ten samples produce eight different answers. On the second, ten samples produce the same wrong answer nine times. What does this tell you about the nature of the model's error in each case, and which error is self-consistency more likely to fix?
The first question shows high variance: the model is uncertain and its errors are not correlated, so the correct answer may well appear in a larger sample and win by plurality. Self-consistency is likely to help here. The second question shows low variance: the model is confidently wrong, and nearly all samples converge on the same incorrect answer. This suggests a systematic bias or a gap in the model's knowledge, and self-consistency will not fix it—it will simply return the wrong answer with high confidence. The distinction matters for deciding whether to sample more or to try a different approach entirely, such as rephrasing the question or providing additional context.
Go deeper
- Self-Consistency Improves Chain of Thought Reasoning in Language Models · arXiv · Xuezhi Wang et al. · 2022-03-21
- Tree of Thoughts: Deliberate Problem Solving with Large Language Models · arXiv · Shunyu Yao et al. · 2023-05-17
Image: Original diagram, The Daily Triptych. Licence: Original work. Source.