Since 2024 the strongest models write a long working-out before answering hard questions, sometimes thousands of tokens of trying, checking and backtracking. These reasoning models are trained with reinforcement learning on problems whose answers can be checked automatically. This module explains why writing out steps helps a transformer, how answer-time compute can substitute for model size, how the training works, and what goes wrong. The mechanisms are well established; numbers from particular models are left out because they date fast.

22.1 Chain-of-Thought

Demand an immediate answer to a multi-step arithmetic question and a model often fails; ask it to "think step by step" and it often succeeds. Writing intermediate steps first, (Wei et al., 2022), helps for a structural reason.

A transformer does a fixed amount of computation per token, one pass through its $L$ blocks, so a question needing more sequential steps than one pass cannot be answered in one token, however large the model. But each written token is fed back as input (section 12.1) for later tokens to attend to. Generating more tokens buys more computation, with the written text as working memory between passes.

The steps also break a hard prediction into easy ones that look like familiar text: worked solutions, derivations, code.

Chain-of-thought can be prompted, by asking or by showing worked examples (section 14.3). Reasoning models are to produce long chains on their own (section 22.3).

22.2 Spending Compute at Answer TimeRun this section

So a model can trade compute at answer time (test-time compute) for accuracy:

Longer reasoning. Reasoning models often improve steadily with their thinking budget.
Self-consistency. Sample $n$ chains and take a majority vote over their answers (Wang et al., 2023). If the right answer is likelier than any single wrong one, the majority beats a single sample.
Best-of- $n$ with a verifier. A verifier picks among $n$ answers: unit tests for code, an exact checker for maths, or a learned scoring model. With a perfect verifier, success needs only one right sample:

P(\text{success with } n) = 1 - (1 - p)^n

With an imperfect learned verifier, gains flatten and can reverse at large $n$ , as more candidates mean more chances to fool it: reward hacking again (section 21.5).

So model size and answer-time compute are partly interchangeable: on some problems a smaller model with a larger budget matches a larger model answering directly (Snell et al., 2024).

Figure 22.2

Three ways to spend samples at answer time

Accuracy against the number of samples n (log scale), for a model whose single samples are right with probability p.

The imperfect verifier accepts a right answer with probability equal to its reliability and a wrong one with the remainder, and takes the first accepted. Majority voting assumes wrong answers are spread over four alternatives.

Predict, then try

Each sample is right with probability 0.3. Before moving the slider, estimate how many samples are needed for best-of- $n$ with a perfect verifier to succeed 90% of the time, and what majority voting does at the same $p$ .

In words. Accuracy against samples, from 1 to 64, for best-of-n with a perfect verifier, best-of-n with an imperfect one, and majority voting. Sliders set per-sample accuracy and verifier reliability.

Browser lab

Majority voting over sampled answers

A simulated solver answers 400 two-digit additions; each sampled answer is right with probability 0.6, and otherwise off by one of four typical slips. Self-consistency takes the most common answer over k samples. Section 22.2 predicts accuracy rises with k when single samples are usually right and wrong answers disagree with each other.

A simulated solver, not a language model: it isolates the statistics of voting. Real samples from one model are correlated (they share its blind spots), which makes the gains smaller than for independent samples.

22.3 RL With Verifiable Rewards and GRPORun this section

Reasoning models are made by RL with verifiable rewards (RLVR) on problems a program can check: mathematics with known answers, code with unit tests, verifiable puzzles. The reward is often 1 for a correct final answer and 0 otherwise, sometimes plus a little for the required format. With no reward model, only the checker itself can be hacked.

The model is never shown how to reason. The policy gradient (section 21.2) simply makes whatever reasoning led to correct answers more likely, and long chains, self-checking ("wait, let me verify that") and backtracking emerge because they raise the reward (DeepSeek-AI, 2025).

GRPO

Group Relative Policy Optimization (GRPO, Shao et al., 2024) replaces PPO's value model (section 21.3) with something cheaper: sample a group of $G$ responses per prompt and use the group's own statistics as the baseline:

A_i = \frac{r_i - \operatorname{mean}(r_1, \dots, r_G)}{\operatorname{std}(r_1, \dots, r_G)}

introduces 1 notation

Every token of response $i$ gets advantage $A_i$ , positive if it beat its siblings, and the update is PPO's clipped objective plus a KL penalty. Dropping the value model and its training matters at the scale of long reasoning chains.

A prompt that every sample gets right (or wrong) has all advantages zero and teaches nothing, so learning concentrates at the edge of the model's ability, and reasoning training sets are often filtered to that difficulty.

Figure 22.3

Group-relative advantages

Eight sampled answers to one problem. Click a tile to mark it correct (reward 1) or incorrect (0). The bars are each answer’s advantage.

Advantage = (reward − group mean) ÷ group standard deviation. Every token of an answer receives its answer’s advantage.

Predict, then try

A group of eight answers to one problem has rewards [1, 0, 0, 1, 0, 0, 0, 0]. Before computing, predict the sign and rough size of the advantage for a correct answer, and what happens if all eight are correct.

In words. A group of answers as a row of tiles marked correct or incorrect, toggled by clicking, each with its group-relative advantage drawn as a bar above or below zero.

22.4 Outcome and Process Rewards

An outcome reward, on the final answer alone, is easy to compute and hard to game but sparse: a 5,000-token chain earns one bit, a chain right until its last line scores the same 0 as one wrong from the start, and a right answer from flawed reasoning is rewarded.

A process reward scores each step. Lightman et al. (2023) trained a process reward model on human step labels and found it a better verifier than an outcome-only one on hard mathematics. But step labels are expensive, and a learned step-scorer can itself be exploited, so large-scale reasoning training mostly uses outcome rewards, with process supervision used for verification and data selection.

22.5 Distilling Reasoning

RLVR is expensive and works best on large models. Small models inherit much of it by sequence-level distillation (section 20.4): fine-tune on a large reasoning model's chains that reach correct answers. DeepSeek-AI (2025) found this beat training the same small models with RL directly, at a fraction of the cost. Needing only the teacher's text is part of why reasoning spread quickly to open models.

22.6 Failure Modes

Unfaithful reasoning. Altering a chain or adding a misleading hint shows models sometimes answer for reasons their stated reasoning omits (Turpin et al., 2023). A chain of thought is evidence about the reasoning, not a transcript of it.
Overthinking. Thousands of tokens on easy questions, sometimes talking itself out of a correct answer. Answer-time compute is paid on every query.
Reward hacking, again. A weak checker can be satisfied without solving the problem, by code that special-cases the tests, say. The checker defines what the model learns, flaws included.
Narrow transfer. Skills trained on mathematics and code transfer only partly to domains with no checker, such as open-ended writing.

Exercises

Check

Why can writing intermediate tokens let a transformer solve problems it cannot answer in one token?

Apply

Each sample is correct with probability 0.4. With a perfect verifier, what is the success rate of best-of-5?

Apply

A GRPO group of four responses has rewards $[1, 1, 0, 0]$ . What is the advantage of a correct response (use the population standard deviation)?

Apply

Why does GRPO need no value model, and what does a prompt whose group is all correct contribute to the update?

Extend

A team's verifier for a coding task runs three unit tests. After RL training, the pass rate rises sharply, but users report that programs often fail on their own inputs. What probably happened, and how would you fix it?

Core Academic References

Wei et al. (2022): Chain-of-Thought Prompting Elicits Reasoning in Large Language Models. NeurIPS 2022. arXiv:2201.11903
Wang et al. (2023): Self-Consistency Improves Chain of Thought Reasoning in Language Models. ICLR 2023. arXiv:2203.11171
Cobbe et al. (2021): Training Verifiers to Solve Math Word Problems. arXiv:2110.14168
Lightman et al. (2023): Let's Verify Step by Step (process reward models). ICLR 2024. arXiv:2305.20050
Snell et al. (2024): Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters. arXiv:2408.03314
Shao et al. (2024): DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models (GRPO). arXiv:2402.03300
DeepSeek-AI (2025): DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning. arXiv:2501.12948
Turpin et al. (2023): Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting. NeurIPS 2023. arXiv:2305.04388
Merrill & Sabharwal (2024): The Expressive Power of Transformers with Chain of Thought. ICLR 2024. arXiv:2310.07923