Chain of thought

Why intermediate steps help, mechanically

A model gets one forward pass per output token, and each pass can only do a bounded amount of computation — roughly proportional to the network’s depth. Asked for a final answer immediately, a hard multi-step problem has to be solved within that one bounded pass, with nowhere to externalise intermediate results. Asked to lay out its working first, the model gets as many passes as the working is long, and each step’s output becomes part of the input to the next step — effectively giving the model a scratchpad it can read back from, spending far more total computation on the problem than a single pass allows. This is the entire mechanical explanation for why “think step by step” helps: it is not that the model becomes smarter, but that it is given more computation to spend, visibly, on a hard problem. It follows directly that chain of thought does not help on tasks that are not actually multi-step — simple lookups, direct classification — because there was no extra computation to usefully spread out in the first place, and in those cases asking for reasoning mostly just adds latency and cost.

Self-consistency, worked through

Self-consistency improves on a single chain of thought by sampling several independent reasoning paths at a non-zero temperature and taking the answer most paths agree on. A toy illustration: asked an arithmetic word problem five times at temperature 0.7, a model might reach the correct answer along three different lines of reasoning, and two incorrect answers via two different mistaken paths. Majority voting returns the correct answer, because the paths that reach it, despite differing in their specific wording, agree with each other, while the wrong answers each arrived by an independent mistake and so do not reinforce one another. This works because errors in reasoning tend to be more varied than correct reasoning is — there are usually many ways to go wrong and comparatively few ways to go right — and it costs roughly the sampled count times as much as a single call, which is why it stayed mostly a research technique rather than a default production pattern.

Reasoning models: training replaces prompting

The biggest change since chain-of-thought prompting was introduced is that some current models are trained with reinforcement learning specifically to produce internal, extended reasoning before answering, rewarded when that reasoning leads to a verifiably correct result — a technique sometimes called reinforcement learning from verifiable rewards. These models generate their own internal deliberation, often lengthy, before committing to a final answer, and typically expose a way to control how much computation to spend on that deliberation — a reasoning-effort setting rather than a prompt instruction. For these models, adding “think step by step” to the prompt is at best redundant, since the behaviour is already trained in, and can occasionally interfere by competing with the model’s own learned strategy for how to structure its reasoning.

Extensions beyond a single chain

Tree-of-thought and similar techniques generalise the idea further, exploring several reasoning branches and evaluating or backtracking between them rather than committing to one linear chain — closer in spirit to a search algorithm than to a single stream of thought. These see use mainly where a problem has a cheap way to check whether a branch is promising partway through, since the exploration cost grows quickly without that.

Never trust the chain as an explanation

The strongest caution this lesson carries: generated reasoning reads like an explanation and is not reliably one. Research probing models that produce a chain of thought has found cases where the model reaches an answer for reasons its own stated reasoning never mentions — the chain is a plausible narrative generated alongside the answer, not necessarily an accurate causal account of how the answer was actually reached internally. If a user-facing product needs to justify an answer, the retrieved sources and the tool results behind it are real evidence; the model’s narrated reasoning is, at best, a helpful hint and should never be presented as proof.