II · THE IDEA · ARTIFICIAL INTELLIGENCE
Chain of Thought
▶ Listen · narrated
A model that answers incorrectly in silence will often answer correctly if you let it write a few sentences first. The improvement is large enough to matter commercially.
At a glance
- What it is
- Prompting a model to generate intermediate reasoning steps before producing a final answer
- Typical gain
- Accuracy on multi-step reasoning tasks often doubles or more compared to direct answering
- Simplest form
- Append "Let's think step by step" to the prompt (Kojima et al., zero-shot)
- Cost trade-off
- Requires generating more tokens per query, increasing latency and expense
Imagine you are trying to solve a long division problem in your head. If you have to give the answer immediately, you might guess or lose track halfway through. But if someone lets you write down the intermediate steps — the first digit of the quotient, the subtraction, the next digit — you are much more likely to get it right, because each written step reminds you where you are and what to do next.
Chain-of-thought prompting works similarly. Instead of asking a model for an answer in one go, you ask it to write out its reasoning first. The model generates sentences that look like worked solutions: "First, I need to find X. Then I use X to calculate Y." Those sentences become part of the context, and they guide the model toward a better final answer. On many tasks, this doubles or triples accuracy compared to answering directly. The simplest version is just adding "Let's think step by step" to your question.
Chain-of-thought prompting is a technique where a model generates intermediate reasoning steps as part of its output before producing a final answer. It can be applied few-shot, by including examples of step-by-step solutions in the prompt, or zero-shot, by appending a phrase like "Let's think step by step" to the question. The method improves accuracy on multi-step reasoning tasks — arithmetic, logical inference, commonsense problems — often by a factor of two or more compared to direct answering.
The mechanism is straightforward in autoregressive terms. Each generated token conditions the distribution for the next token. When a model must produce an answer immediately, all intermediate reasoning is compressed into the forward passes that generate that answer token and any brief preamble. When allowed to generate steps, the model spreads the reasoning across many tokens. Early tokens can surface relevant facts, frame sub-problems, or activate useful patterns in the weights, and those tokens remain in the context window, available to influence later generation. The written chain is not separate from generation; it is generation, but structured to improve the odds of correctness.
Importantly, the written steps are not a transcript of computation. The model does not execute the arithmetic or logic you see. It produces text that resembles worked solutions from training data, and that text steers subsequent tokens, but the relationship is statistical. A chain can contain errors and still lead to a correct answer if it activates the right weights, and a flawless chain can lead to a wrong answer if the final token is sampled from a distribution that still assigns non-trivial probability to incorrect options. The steps are a scaffold, not a proof.
The effect scales with model size. Wei and colleagues found that chain-of-thought prompting produced little benefit in models below tens of billions of parameters. Smaller models would generate plausible-looking intermediate text that did not improve, and sometimes hurt, accuracy. Only above a certain scale did the technique reliably help. This suggests the method does not compensate for lack of capacity; it exploits capacity that is already present but underutilised by direct prompting.
Cost is non-trivial. Generating a chain of reasoning produces many more tokens per query than direct answering, increasing latency and expense. In production systems, this trade-off is managed by using chain-of-thought selectively: on tasks known to benefit, or when accuracy matters more than speed, or by generating chains for several candidate answers and selecting the most common final result (self-consistency decoding).
Look closer
The steps are not necessarily the reasoning
The written chain is generated token by token, left to right, like any other output. Each step influences the next, and the final answer, but the model does not execute the arithmetic or logic you see written down. It produces text that resembles worked solutions from its training data. Sometimes the written steps contain errors but the final answer is correct, or vice versa. The chain is a scaffold that changes the distribution the model samples from, not a transcript of internal computation.
Few-shot and zero-shot variants both work
Wei and colleagues showed the technique by providing several examples of problems solved step-by-step in the prompt, then asking the model to solve a new problem the same way. Kojima and colleagues found that simply appending the phrase "Let's think step by step" to a question, with no examples at all, produced similar gains on many tasks. The zero-shot version is simpler and does not consume context window with examples, but the few-shot version can guide the format and style of reasoning more tightly.
The gain scales with model size
Chain-of-thought prompting produces little or no improvement in small models. In Wei's experiments, the effect became pronounced only above a certain parameter count — roughly tens of billions of parameters, though the threshold varies by task. Below that scale, models given space to write steps often wrote plausible-looking text that did not help, or even hurt, final accuracy. The technique is not a substitute for scale; it is something scale unlocks.
The story
In 2022, two groups published closely timed papers showing that letting a language model write out intermediate steps before answering a question could dramatically improve accuracy on tasks that required multiple logical moves. Wei and colleagues at Google demonstrated the effect by including several worked examples in the prompt, each showing a problem solved step-by-step. When the model then encountered a new problem, it would generate its own chain of reasoning and arrive at the correct answer far more often than if asked to respond directly. Kojima and colleagues showed that explicit examples were not always necessary: adding the single phrase "Let's think step by step" after a question produced similar gains, turning the technique into something you could apply to any query without preparation.
The improvement was not small. On some arithmetic and logical reasoning benchmarks, accuracy doubled or tripled. A model that answered twenty per cent of questions correctly in direct mode might answer sixty per cent correctly when allowed to reason aloud. The effect held across a range of tasks: grade-school maths problems, commonsense reasoning, symbolic manipulation. It did not work universally — some questions benefited more than others — but the pattern was consistent enough to matter.
The mechanism is not mysterious in outline. A model generates one token at a time, and each token conditions what comes next. When forced to produce an answer immediately, the model must compress all intermediate reasoning into the small number of forward passes that happen during generation of that answer. When allowed to write steps, it spreads the work across more tokens. Early steps can surface relevant facts, frame sub-problems, or rule out wrong paths, and those tokens then sit in the context, available to influence later ones. The written chain is not a separate reasoning process; it is part of the same autoregressive generation, but it gives the model more opportunities to correct course.
What the written steps are not is a faithful record of internal computation. The model does not perform arithmetic by manipulating symbols the way a calculator does. It produces text that looks like worked solutions because such text appeared in its training data, and that text, once generated, steers subsequent tokens toward better answers. Sometimes a model will write a step that contains an arithmetic error but still produce the correct final answer, suggesting the written work and the effective computation are not the same thing. Other times the steps are flawless but the answer is wrong. The chain is a scaffold, not a transcript.
The technique also revealed something about scale. In smaller models, writing steps helped little or not at all. Below a certain size — roughly tens of billions of parameters, though this varied by task — models would produce plausible-sounding intermediate text that did not improve, and sometimes worsened, their final answers. Only in larger models did the written chain reliably guide generation toward correctness. Chain-of-thought prompting is not a trick that compensates for lack of capacity. It is a trick that capacity makes possible.
Why it mattered then
The 2022 papers arrived at a moment when the largest language models were beginning to show competence on tasks that required more than retrieval or pattern completion. They could answer factual questions, translate, summarise. But they struggled with anything that required holding several pieces of information in mind and combining them in sequence: multi-step arithmetic, logical puzzles, planning. The gap between what models could do and what people wanted them to do was widening. Chain-of-thought prompting mattered because it closed part of that gap without retraining. You did not need new architectures, new data, or months of compute. You needed only to change how you wrote the prompt. For research groups with access to large models but not the resources to train new ones, and for companies trying to build products on top of APIs, this was immediately useful. It also mattered scientifically. The fact that writing steps helped at all suggested that the models had latent capabilities that direct prompting failed to elicit. The fact that the effect scaled with size suggested something about how those capabilities emerged. It was evidence that scale was not just making models better at the same things; it was making qualitatively different things possible.
Why it matters now
Chain-of-thought prompting is now a standard tool. Most commercial and open-weight models are deployed with system prompts or fine-tuning that encourages step-by-step reasoning on appropriate tasks. The zero-shot variant — "Let's think step by step" or similar phrasings — is simple enough that people use it casually, often without knowing they are applying a named technique. The few-shot variant is common in settings where output format matters: customer support, code generation, structured decision-making. It also set off a line of research. If writing steps helps, what about writing more steps, or different kinds of steps? What if you generate several chains and pick the answer that appears most often? What if you search over possible chains, or let the model critique and revise its own reasoning? These elaborations — tree-of-thought, self-consistency, iterative refinement — all descend from the 2022 observation that intermediate text changes what a model can do. The technique also clarified a limitation. The written steps are not the reasoning. They are text that improves the odds of correct text coming next. This matters for interpretability: you cannot trust a chain-of-thought output as an explanation of why the model produced its answer, even though it looks like one. It also matters for reliability: a model can write convincing-looking steps and still be wrong, because the steps are themselves generated and fallible. Chain-of-thought prompting makes models more accurate, but it does not make them transparent.
The surprising detail
Sometimes a model writes a chain of reasoning that contains a clear factual or arithmetic error, but then produces the correct final answer anyway. Other times the written steps are impeccable and the answer is wrong. This asymmetry is evidence that the chain is not the computation. The model is not executing the steps you see; it is producing tokens that resemble worked solutions, and those tokens influence what comes next, but the relationship between written work and correctness is statistical, not logical. A step can be wrong and still steer the model toward a right answer if it activates the right patterns in the weights. Conversely, a perfect chain can fail if the final answer token is drawn from a distribution that still assigns meaningful probability to wrong answers. The written reasoning is a prompt to itself, not a proof.
Remember this
Letting a model write steps before answering improves multi-step accuracy substantially, but the steps are generated text that influences the next token, not a transcript of computation.
Test yourself
You are building a system that uses chain-of-thought prompting to solve maths problems. A user reports that the model sometimes writes a correct solution but then gives a wrong final answer. You consider hiding the chain from the user and showing only the final answer. Name two distinct reasons this might be a bad idea.
First, the written chain, even when the final answer is wrong, often reveals where the reasoning went astray, which helps users decide whether to trust the output or try again with a clearer prompt. Hiding it removes that signal. Second, in some deployments you may want to log or audit the reasoning for safety, compliance or debugging, and the chain is the only window you have into what patterns the model activated, even though it is not a faithful transcript. A third reason: users often learn to write better prompts by seeing what kind of intermediate steps the model produces, and hiding the chain removes that feedback loop. The chain is not the reasoning, but it is still informative.
Go deeper
- Chain-of-Thought Prompting Elicits Reasoning in Large Language Models · arXiv · Jason Wei et al. · 2022-01-28
- Large Language Models are Zero-Shot Reasoners · arXiv · Takeshi Kojima et al. · 2022-05-24
Image: Original diagram, The Daily Triptych. Licence: Original work. Source.