Instruction tuning (section 20.1) teaches by imitation, so it cannot make a model better than its examples, which are expensive to write. It is far easier for people to say which of two answers is better. This module turns such judgments into training, using a small piece of reinforcement learning (RL), which learns from scores rather than correct answers, then builds the reward model, the RLHF loop, and DPO, the shortcut that removes most of that machinery. Module 22 uses the same tools to train reasoning.

21.1 A Reinforcement Learning Primer

An agent takes actions in an environment, each changing the state it sees next.
A policy is its rule for acting: a distribution over actions given the state.
A reward scores an outcome. An episode runs from start to finish, and its return $R$ is its total reward.
The goal is to adjust the policy to maximize the expected return.

Each has a language-model counterpart:

RL	language model
state	the prompt plus the tokens generated so far
action	the next token
policy	the model's next-token distribution
episode	generating one complete response $y$ to a prompt $x$
reward	a score for the finished response, usually given once at the end

So a language model already is a policy: $\pi_\theta(y \mid x)$ is the product of its next-token probabilities (section 3.3). The objective is the expected reward of its responses:

J(\theta) = \mathbb{E}_{x \sim \mathcal{D},\; y \sim \pi_\theta(\cdot \mid x)}\left[R(x, y)\right]

introduces 2 notations

$y \sim \pi_\theta(\cdot \mid x)$ reads "the response is sampled from the model". Unlike supervised learning, there is no correct answer to imitate, only a score for whatever the model produced, so it must explore and learn from how its responses scored.

21.2 Policy GradientsRun this section

$J$ averages over responses sampled from the model itself, and sampling has no derivative. The log-derivative trick, , gets around it:

\nabla_\theta J = \mathbb{E}_{y \sim \pi_\theta}\left[R(x, y)\, \nabla_\theta \log \pi_\theta(y \mid x)\right]

This policy gradient is estimated from samples by REINFORCE (Williams, 1992): sample a response, score it, and step toward making it more likely, scaled by its reward. $\nabla_\theta \log \pi_\theta(y \mid x)$ is the gradient of the response's log-probability, which backpropagation computes as in training.

The estimate is right on average but very noisy: if every reward is positive, every sample is pushed up, and only the relative sizes carry information. The fix subtracts a baseline $b$ , an estimate of the typical reward:

\nabla_\theta J = \mathbb{E}_{y \sim \pi_\theta}\left[\big(R(x, y) - b\big)\, \nabla_\theta \log \pi_\theta(y \mid x)\right]

The average gradient is unchanged, since , but the noise falls: better-than-usual responses are pushed up and worse ones down. $A = R - b$ is the advantage.

Figure 21.2

REINFORCE on a three-armed bandit, with and without a baseline

Three actions with average rewards 5.0, 5.5 and 6.0 (plus noise): all positive, and close together relative to their size. Every line is one action’s probability over 400 steps; several independent runs are overlaid.

Learning rate 0.02 on the logits. The baseline is a running average of recent rewards.

Predict, then try

The simplest RL problem: three actions with fixed average rewards 5.0, 5.5 and 6.0, all positive. Run REINFORCE without a baseline, then with one. Before you do, predict which version converges to the best action faster and more reliably.

In words. A three-armed bandit (one state, three actions with noisy rewards around fixed means) trained with REINFORCE, with and without a baseline. The chart traces each action's probability over training, and several runs can be overlaid to compare their spread.

21.3 PPORun this section

At language-model scale, one large step can change the policy so much that its samples no longer describe it, and training collapses. Proximal Policy Optimization (PPO, Schulman et al., 2017) limits each update. It takes several gradient steps on each sampled batch, comparing each token's new probability with the sampling policy's, $\rho = \pi_\theta(a \mid s) / \pi_{\text{old}}(a \mid s)$ , in a clipped objective:

\mathcal{L}_{\text{PPO}} = \mathbb{E}\left[\min\left(\rho\, A, \; \operatorname{clip}(\rho, 1 - \epsilon, 1 + \epsilon)\, A\right)\right]

introduces 2 notations

With $\epsilon$ typically 0.2, inside $[0.8, 1.2]$ this is the advantage-weighted policy gradient; once a good action is 20% more likely (or a bad one 20% less), the clip removes the incentive to go further on this batch. The policy moves in many small, safe steps.

For language models, PPO adds a value model, a copy of the language model with a scalar output that predicts the expected reward from each partial response and supplies a per-token baseline, and a KL penalty toward the starting model (section 21.5). That is four models in memory at once (policy, reference, reward, value), the main reason simpler methods were sought.

21.4 Reward Models From PreferencesRun this section

A chat response's reward has no formula, so RLHF, as used for InstructGPT (Ouyang et al., 2022), learns it from people. They compare pairs of responses to a prompt $x$ , a winner $y_w$ and a loser $y_l$ , and a $r_\phi$ , usually a copy of the language model with a scalar output, is trained to score the winner higher:

\mathcal{L}_{\text{RM}} = -\log \sigma\left(r_\phi(x, y_w) - r_\phi(x, y_l)\right)

introduces 1 notation

This is the Bradley–Terry model of section 3.6 fitted by maximum likelihood: $\sigma(r_w - r_l)$ is the predicted probability of preferring $y_w$ , and only differences of reward are meaningful.

21.5 RLHF End to End

Then PPO trains $\pi_\theta$ toward responses the reward model scores highly, without drifting far from the instruction-tuned model $\pi_{\text{ref}}$ it started from:

\max_{\theta} \; \mathbb{E}_{x, \, y \sim \pi_\theta}\left[r_\phi(x, y)\right] - \beta \, \operatorname{KL}\left(\pi_\theta(\cdot \mid x) \,\|\, \pi_{\text{ref}}(\cdot \mid x)\right)

introduces 3 notations

$\beta$ sets how strongly drift is penalized; in practice $-\beta \log \frac{\pi_\theta}{\pi_{\text{ref}}}$ is added to the reward token by token. The penalty matters: the reward model only approximates human judgment, and a policy optimized against it without restraint finds outputs that score well and read badly, reward hacking. The result was striking: people preferred the 1.3B-parameter InstructGPT's answers to the 175B-parameter GPT-3's.

21.6 Direct Preference OptimizationRun this section

RLHF's reward model, sampling and RL loop are all hard to tune. Rafailov et al. (2023) showed that the RLHF optimum has a closed form, which eliminates the reward model, in three steps.

The best policy has a closed form. For a fixed reward $r$ , section 21.5's objective is maximized by reweighting the reference model by the exponentiated reward:

\pi^*(y \mid x) = \frac{1}{Z(x)}\, \pi_{\text{ref}}(y \mid x)\, \exp\left(\frac{r(x, y)}{\beta}\right)

introduces 1 notation

where $Z(x)$ makes the probabilities sum to 1. Responses the reward likes become more likely, by a factor $\beta$ controls; responses the reference never produces stay unlikely. 2. So the reward can be read off the policy: $r(x, y) = \beta \log \frac{\pi^*(y \mid x)}{\pi_{\text{ref}}(y \mid x)} + \beta \log Z(x)$ . Any policy implicitly defines the reward for which it is optimal. 3. Substitute into Bradley–Terry. Preferences depend only on the difference of two rewards for the same prompt, so $\beta \log Z(x)$ cancels, leaving only the policy and the reference model.

Maximum likelihood on the preferences, with $\pi_\theta$ standing in for $\pi^*$ , gives a loss on the pairs themselves:

\mathcal{L}_{\text{DPO}} = -\log \sigma\left(\beta \log \frac{\pi_\theta(y_w \mid x)}{\pi_{\text{ref}}(y_w \mid x)} - \beta \log \frac{\pi_\theta(y_l \mid x)}{\pi_{\text{ref}}(y_l \mid x)}\right)

Each log-ratio measures how much more likely the model has made a response than the reference did; the loss pushes the winner's up and the loser's down. ( $\beta$ plays its RLHF role, unrelated to Adam's $\beta_1$ and $\beta_2$ .) It is ordinary supervised training on a fixed dataset, which is why DPO and its variants are widely used alongside or instead of RLHF.

Browser lab

DPO recovers the closed-form optimal policy

One prompt with five possible responses, a hidden reward for each, and a reference policy. 4,000 preference pairs are sampled from the Bradley–Terry model under the hidden reward, and a policy is trained on them with the DPO loss (β = 0.5) and nothing else: no reward model, no sampling. Section 21.6 predicts it ends up at π* ∝ π_ref · exp(r/β).

A single prompt and a policy that can represent any distribution over five responses, so the only error left is from the finite sample of preferences. Real models share parameters across prompts and cannot reach the optimum exactly.

Exercises

Check

Map these RL terms onto a language model: state, action, policy, episode.

Apply

A reward model scores the preferred answer 2.0 and the other 0.5. What is $\mathcal{L}_{\text{RM}}$ for this pair?

Apply

In PPO with $\epsilon = 0.2$ and a positive advantage $A = 1$ , what is the clipped objective's value for a token whose probability ratio is $\rho = 1.5$ ?

Apply

Derive the optimal policy $\pi^* \propto \pi_{\text{ref}} \exp(r/\beta)$ for a single prompt with two possible responses, by maximizing $p\,r_1 + (1-p)\,r_2 - \beta \operatorname{KL}$ , where $p$ is the probability of response 1 and the reference gives it $q$ .

Extend

Explain reward hacking with an example, and say which part of the RLHF objective guards against it.

Core Academic References

Sutton & Barto (2018): Reinforcement Learning: An Introduction (2nd ed.). MIT Press.
Williams (1992): Simple Statistical Gradient-Following Algorithms for Connectionist Reinforcement Learning (REINFORCE). Machine Learning, 8: 229–256.
Schulman et al. (2017): Proximal Policy Optimization Algorithms. arXiv:1707.06347
Christiano et al. (2017): Deep Reinforcement Learning from Human Preferences. NeurIPS 2017. arXiv:1706.03741
Ouyang et al. (2022): Training language models to follow instructions with human feedback (InstructGPT). NeurIPS 2022. arXiv:2203.02155
Rafailov et al. (2023): Direct Preference Optimization: Your Language Model is Secretly a Reward Model. NeurIPS 2023. arXiv:2305.18290
Gao, Schulman & Hilton (2023): Scaling Laws for Reward Model Overoptimization. ICML 2023. arXiv:2210.10760