II · THE IDEA · ARTIFICIAL INTELLIGENCE
Reasoning Models
▶ Listen · narrated
Some models now pause to think for thirty seconds, generating reasoning you can read. The delay is real computation, not theatre — but the engineers cannot always explain what the model decided to think about.
At a glance
- What they do
- Generate long chains of reasoning tokens before producing a final answer
- Training signal
- Reward based on whether the final answer is correct, not on the reasoning steps themselves
- Typical reasoning length
- Thousands to tens of thousands of tokens, often hidden from the user
- Key unknown
- Whether the reasoning is faithful to the model's actual process or a post-hoc rationalisation
Imagine you are learning to solve mathematics problems, and someone rewards you only when your final answer is correct. They never tell you whether your working is good or bad, only whether you got the right number at the end. You would probably start writing out steps, because that helps you keep track. But the steps you write might not be the whole story of how you thought about the problem. You might do some thinking in your head and only write down the parts that look like proper working. Reasoning models are trained the same way. They generate long sequences of tokens before they answer, and those sequences often look like step-by-step problem solving. But the model is rewarded only for correct final answers, not for clear reasoning. The reasoning trace is whatever helped the model get to the right answer, and it may not be a complete or honest record of the model's thought process. It is more tokens, generated by the same mechanism, but with no external supervision to ensure they mean what they appear to mean.
Reasoning models are language models trained with reinforcement learning to generate extended reasoning traces before producing a final answer. The training signal is typically a reward based on outcome correctness, not on the quality of intermediate steps. In DeepSeek-R1, the model generates a chain of thought and then an answer; if the answer is correct, the entire sequence receives a positive reward under a policy gradient method. There is no step-by-step verification, no process supervision, and no explicit loss term penalising unfaithful or illegible reasoning. The model learns to allocate compute across tokens in whatever way maximises the probability of correct answers, and structured reasoning emerges as a learned strategy rather than an imposed constraint. This approach scales better than training separate verifier models for each reasoning step, as earlier work by Lightman et al. required, but it provides no guarantee of faithfulness. The reasoning trace is causally prior to the answer — the tokens are generated in sequence, so earlier tokens do condition later ones through the attention mechanism — but this does not mean the trace is a complete record of the computation. The model may be performing reasoning that is not visible in the token stream, or it may be generating a post-hoc narrative that fits the training distribution. The trace is also often hidden from the user in deployed systems, with only a summary or the final answer exposed. This makes auditing difficult: you cannot determine from the output alone whether an error arose from faulty reasoning, missing knowledge, or a mismatch between the model's actual process and the trace it produced.
Look closer
The training uses outcome rewards, not step verification
DeepSeek-R1 is trained with reinforcement learning that rewards the model only when the final answer is correct. There is no human annotation of the reasoning steps, no explicit supervision telling the model that line 47 of its internal monologue is good thinking and line 48 is confused. The model learns to produce whatever chain of tokens makes correct answers more likely, but the training signal does not directly shape the content of that chain. This is computationally cheaper than the alternative, which involves training separate verifier models to score each reasoning step, as described in earlier research by Lightman and colleagues.
The reasoning is often hidden from the user
Many reasoning models generate their full chain of thought internally but show you only a summary or the final answer. OpenAI's o1 produces thousands of reasoning tokens but displays a condensed version. The full trace exists — the model genuinely performed that computation — but you do not see it, partly for speed and partly because the raw output can be repetitive or hard to follow. This creates an asymmetry: the model's behaviour is shaped by a process you cannot audit, and you cannot easily tell whether a wrong answer came from faulty reasoning or faulty knowledge.
The model invents its own reasoning format
Because the training signal comes only from final-answer correctness, the model is free to develop whatever reasoning style helps it succeed. DeepSeek-R1 was not told to use structured problem decomposition or to check its work. Those behaviours emerged during training because they correlate with correct answers. The researchers report that the model sometimes produces chains that look like human step-by-step reasoning, but also chains that do not, and there is no guarantee the visible reasoning is the actual cognitive path. It may be a narrative the model learned to tell because such narratives accompany correct answers in its training data.
The story
A reasoning model is a language model trained to generate a long sequence of tokens — its reasoning trace — before it produces the answer you asked for. The trace is not a separate module or a different kind of computation. It is more tokens, generated by the same mechanism that writes everything else, but the model has learned that writing them first makes the final answer more likely to be correct.
The training method, at least as described in the DeepSeek-R1 paper, is reinforcement learning with a reward signal based on outcomes. The model generates a reasoning trace and then an answer. If the answer is correct, the entire sequence — trace plus answer — receives a positive reward. If the answer is wrong, the sequence receives a penalty. The model adjusts its weights to make rewarded sequences more probable. Crucially, the reward is not given to individual reasoning steps. There is no human annotator marking step three as good logic and step seven as a mistake. The signal comes only from the final answer, and the model must work out for itself what kind of intermediate thinking leads there.
This is cheaper and more scalable than the alternative, which is to train a separate verifier model that scores each step of reasoning, as Lightman and colleagues described in earlier work. Step-by-step verification requires detailed human judgement on many reasoning traces, which is expensive and slow. Outcome-based training needs only a correct answer, which for many tasks can be checked automatically. The trade-off is that you lose direct control over what the reasoning looks like. The model learns whatever helps, and that may not be what you expected.
DeepSeek-R1's researchers report that the model developed recognisable reasoning patterns without being told to. It breaks problems into parts. It checks its work. It sometimes backtracks when it notices an error. These behaviours were not programmed; they emerged because they correlate with correct answers. But the same training process can also produce reasoning traces that are harder to interpret, or that do not resemble human problem-solving at all. The model is optimising for outcomes, not for legibility.
There is also a question the training method does not answer: is the reasoning trace faithful? Does it reflect the model's actual cognitive process, or is it a plausible story the model learned to tell because such stories accompany correct answers in the training data? The model generates the trace causally — the reasoning tokens come before the answer tokens, so they do influence what the answer will be — but that does not mean the trace is a transparent record of how the model arrived at the answer. It may be doing some of its reasoning in ways that are not visible in the token stream, and then producing a narrative that fits.
Many reasoning models do not show you the full trace. OpenAI's o1 generates thousands of reasoning tokens internally but displays only a summary. The full computation happened — the model spent the time and the tokens — but you see a compressed version. This is partly practical: raw reasoning traces can be long and repetitive. But it also means you cannot easily audit the process. If the model gives a wrong answer, you cannot tell whether the reasoning was sound but the knowledge was faulty, or whether the reasoning itself went astray.
Why it mattered then
Reasoning models represent a shift in how reinforcement learning is applied to language models. Earlier work, including the research by Lightman and colleagues, explored training verifier models to score individual reasoning steps. That approach required detailed human annotation: someone had to read each step of a solution and mark it as correct or incorrect. It worked, but it was expensive and hard to scale. Outcome-based reinforcement learning, as used in DeepSeek-R1, removes that bottleneck. You need only a correct final answer, which for many domains can be verified automatically. This made it feasible to train reasoning behaviour at scale without a proportional increase in human labelling effort. The method also revealed something unexpected: that models could learn structured, multi-step reasoning without explicit supervision of those steps. The reasoning patterns emerged as a side effect of optimising for correct answers, which suggested that the models were discovering general problem-solving strategies rather than memorising specific solution templates.
Why it matters now
Reasoning models matter now because they perform measurably better on tasks that require multi-step thinking, and because they expose a new kind of interpretability problem. The performance gains are substantial on benchmarks involving mathematics, code generation, and logical puzzles — domains where a single forward pass often fails but a chain of reasoning succeeds. But the reasoning traces themselves are hard to trust. You cannot be certain that what you read is the model's actual thought process, and in many deployments you do not see the full trace at all. This creates a practical problem for anyone relying on these systems in high-stakes contexts: the model's behaviour is shaped by a process you cannot fully audit, and the training method offers no guarantee that the reasoning is faithful. At the same time, the success of outcome-based training has made reasoning models cheaper to develop, which means they are likely to become widespread. Understanding what is known and what remains uncertain about how they work is now a basic requirement for anyone deploying or evaluating them.
The surprising detail
DeepSeek-R1 sometimes produces reasoning traces that do not look like human reasoning at all, and the researchers report that these non-human traces can still lead to correct answers. The model is not constrained to think the way a person would. It is optimising for outcomes, and if some other cognitive path works better, the training signal will reinforce that path instead. This means the reasoning trace may not be interpretable even in principle — not because the model is hiding something, but because the problem-solving strategy it discovered does not map onto human concepts. There is also evidence that the model sometimes generates traces that are plausible narratives rather than faithful records. It has learned that certain kinds of reasoning language accompany correct answers, and it reproduces that language even when the actual computation may be happening differently.
What is disputed
It is not known whether the reasoning traces produced by these models are faithful representations of their cognitive process or learned narratives that correlate with correct answers. The training method provides no direct incentive for faithfulness, only for final-answer correctness. The DeepSeek-R1 paper reports that reasoning patterns emerge without explicit supervision, but does not claim that the visible reasoning is the only computation happening, or that it is interpretable in human terms.
Remember this
Reasoning models generate long internal traces before answering, trained by rewarding correct outcomes rather than supervising each step. What the trace actually represents remains uncertain.
Test yourself
A reasoning model gives you a wrong answer and shows you a ten-thousand-token reasoning trace that looks careful and structured. Name two distinct reasons why the trace might not tell you where the error occurred.
First, the trace may not be faithful: the model may be doing some of its reasoning in ways that are not visible in the token sequence, and the trace you see is a plausible narrative rather than a transparent record. If the actual error happened in some non-linguistic part of the computation, the trace will not show it. Second, even if the trace is faithful, the model was trained only on final-answer correctness, so it never learned to distinguish good reasoning steps from bad ones. A step that looks sound to you might be wrong in a way the model cannot recognise, because it never received feedback on individual steps. The training signal does not teach the model to produce legible errors; it teaches it to produce correct answers, and the reasoning is whatever helped get there.
Go deeper
- DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning · arXiv · DeepSeek-AI et al. · 2025-01-22
- Let's Verify Step by Step · arXiv · Hunter Lightman et al. · 2023-05-31
Image: Original diagram, The Daily Triptych. Licence: Original work. Source.