A language model's output is a probability distribution, its training objective is a statement about probabilities, and one of its key design choices (section 6.3) rests on a calculation with averages and spreads. This module teaches the probability the course uses, in the order it uses it. Nothing here is specific to transformers, and every later module leans on it.

3.1 Random Variables, Distributions, and SamplingRun this section

A random variable is a quantity whose value is uncertain until you observe it: the face of a die, the next token of a text, the initial value of a weight. Its distribution says how likely each value is.

For a variable with a list of possible values, the distribution gives each value a probability between 0 and 1, summing to 1. A fair die gives each face $\tfrac{1}{6}$ ; a language model gives one probability to each of its tokens. $P(X = x)$ , or just $p(x)$ , is "the probability that $X$ comes out as $x$ ".

A variable that can take any real value, such as a weight, is described by a density $p(x)$ instead: the probability of landing in a small interval around $x$ is about $p(x)$ times the interval's width, and the total area under $p$ is 1. The most important density is the normal distribution (the bell curve), written :

p(x) = \frac{1}{\sqrt{2\pi\sigma^2}} \exp\left(-\frac{(x - \mu)^2}{2\sigma^2}\right)

introduces 2 notations

It is centred on $\mu$ with width set by $\sigma$ : about 68% of draws land within one $\sigma$ of $\mu$ , and 95% within two. Weights are usually initialized from a normal distribution with $\mu = 0$ and small $\sigma$ (section 10.4). reads " $x$ is drawn from a normal distribution with mean 0 and variance 1".

Sampling draws a value at random according to the distribution. Generating text is sampling, one token at a time (section 12.3).

Two random variables are independent when knowing one tells you nothing about the other: two dice rolls are, the first and second tokens of a sentence are not.

3.2 Expectation and VarianceRun this section

The expectation is the average value of $X$ , weighting each value by its probability. It is also called the mean:

\mathbb{E}[X] = \sum_x x \, p(x)

introduces 1 notation

For a fair die, $\mathbb{E}[X] = \tfrac{1}{6}(1 + 2 + 3 + 4 + 5 + 6) = 3.5$ . (For a density the sum becomes an integral, and the rules below still hold.) The average of many independent samples settles toward the expectation.

The variance is the average squared distance from the mean. Its square root, the standard deviation $\sigma$ , is in the same units as $X$ :

\operatorname{Var}(X) = \mathbb{E}\left[(X - \mathbb{E}[X])^2\right] = \mathbb{E}[X^2] - \left(\mathbb{E}[X]\right)^2

introduces 1 notation

The second form, "the average of the square minus the square of the average", is usually easier to compute.

The Rules

These rules are used repeatedly, most importantly in section 6.3. Here $a$ and $b$ are fixed numbers.

Expectation is linear, always. $\mathbb{E}[aX + bY] = a\,\mathbb{E}[X] + b\,\mathbb{E}[Y]$ , whether or not $X$ and $Y$ are independent.
For independent variables, the expectation of a product is the product of the expectations. $\mathbb{E}[XY] = \mathbb{E}[X]\,\mathbb{E}[Y]$ .
Scaling a variable scales its variance by the square. $\operatorname{Var}(aX) = a^2 \operatorname{Var}(X)$ : doubling every value quadruples the squared distances. Adding a constant leaves the variance unchanged.
For independent variables, variances add. $\operatorname{Var}(X + Y) = \operatorname{Var}(X) + \operatorname{Var}(Y)$ . So a sum of $n$ independent variables with variance $s^2$ each has standard deviation $s\sqrt{n}$ : spread grows like the square root of the number of terms.

By rules 3 and 4, the average of $n$ independent samples has standard deviation shrinking like $1/\sqrt{n}$ . A gradient on a random batch is such an average: by rule 1 its expectation is the full-dataset gradient, and its noise shrinks as the batch grows. A quantity whose expectation equals what it estimates is unbiased.

Figure 3.2

The spread of a dot product grows like the square root of the length

Each sample is the dot product of two random vectors whose entries are drawn from N(0, 1). The dashed curve is the normal distribution with variance d that the rules predict.

4,000 samples per setting. The horizontal axis is fixed at ±48 so the widening is visible.

Predict, then try

The figure draws two random vectors of length $d$ , every entry from $\mathcal{N}(0, 1)$ , and records their dot product. Using rules 2 and 4, predict the standard deviation of the dot product at $d = 64$ .

In words. The figure repeatedly draws two random vectors of a chosen length and adds their dot product to a histogram, over a normal curve with variance equal to the length. The measured standard deviation tracks the predicted square root of the length as the length grows.

3.3 Joint and Conditional Probability

The joint probability $P(A, B)$ is the probability that both $A$ and $B$ happen. The conditional probability , "the probability of $A$ given $B$ ", is the share of $B$ 's probability in which $A$ also happens:

P(A \mid B) = \frac{P(A, B)}{P(B)}

Worked example. In a corpus, the word the makes up 5% of tokens, and the followed by cat makes up 0.01% of adjacent pairs. Then $P(\texttt{cat} \mid \texttt{the}) = 0.0001 / 0.05 = 0.002$ : of all the places a the appears, cat follows in 0.2% of them.

Rearranged, $P(A, B) = P(B)\,P(A \mid B)$ . Applied repeatedly, this gives the chain rule of probability: a joint probability is a product of one-at-a-time conditionals.

P(w_1, w_2, w_3) = P(w_1)\, P(w_2 \mid w_1)\, P(w_3 \mid w_1, w_2)

introduces 1 notation

This is exact, and it is the foundation of every text-writing model in this course: a model that answers "how likely is the next token, given everything before it?" can multiply its answers into the probability of a whole text (section 8.4).

Independence means $P(A \mid B) = P(A)$ , or equivalently $P(A, B) = P(A)\,P(B)$ .

3.4 Maximum LikelihoodRun this section

A model with assigns a probability to each possible observation. The of $\theta$ is the probability it gives the data actually observed, and maximum likelihood chooses the $\theta$ that makes it largest.

For independent observations the likelihood is a product, which underflows in floating point and is awkward to differentiate. The logarithm turns it into a sum, and since the log only increases, maximizing it picks the same $\theta$ . Flipping the sign gives the negative log-likelihood (NLL), a loss to minimize:

\mathcal{L}_{\text{NLL}}(\theta) = -\sum_{i=1}^{n} \log p_\theta(x_i)

introduces 3 notations

Every log of a probability is at most 0, so the loss is positive, and zero only when every observation gets probability 1.

Worked example. A coin with unknown heads-probability $p$ is flipped 10 times and lands heads 7 times. The likelihood is $p^7 (1 - p)^3$ , so the NLL is $-7 \log p - 3 \log(1 - p)$ . Its derivative is $-\frac{7}{p} + \frac{3}{1 - p}$ . Setting it to zero gives $7(1 - p) = 3p$ , so $p = 0.7$ : the maximum-likelihood estimate is the observed frequency.

With many values, value $k$ seen $n_k$ times out of $n$ gets probability $n_k / n$ . A language model is fitted by the same principle, but its probabilities come from a neural network, so there is no formula for the best $\theta$ and gradient descent (section 2.4) does the minimizing.

Browser lab

Maximum likelihood recovers the observed frequency

Flip a coin with a hidden heads-probability, then try every candidate p from 0.001 to 0.999 and keep the one with the lowest negative log-likelihood. Section 3.4 predicts the winner is exactly the fraction of heads observed, not the hidden true value.

flips

Run it several times at 10 flips and then at 5000. The estimate always equals the observed frequency; how close that frequency is to the hidden truth depends on how much data you have.

3.5 Information: Entropy, Cross-Entropy, KL, PerplexityRun this section

The surprise of an outcome with probability $p$ is $-\log p$ : 0 when $p = 1$ , growing without limit as $p$ shrinks. With base-2 logs it is in bits (a fair coin flip carries 1 bit); with natural logs, nats.

The of a distribution is its average surprise: how unpredictable it is.

H(p) = -\sum_x p(x) \log p(x)

A distribution that puts all its probability on one value has entropy 0. A uniform distribution over $K$ values has the largest possible entropy, $\log K$ .

If the true distribution is $p$ but you predict with $q$ , your average surprise, using $q$ 's probabilities over what really happens, is the :

H(p, q) = -\sum_x p(x) \log q(x)

It is never smaller than $H(p)$ , and equals it only when $q = p$ . The excess is the Kullback–Leibler divergence, a measure of how far $q$ is from $p$ :

\operatorname{KL}(p \,\|\, q) = H(p, q) - H(p) = \sum_x p(x) \log \frac{p(x)}{q(x)}

introduces 2 notations

It is never negative, is 0 only when the distributions agree, and is not symmetric: $\operatorname{KL}(p \| q)$ and $\operatorname{KL}(q \| p)$ usually differ. Section 21.5 uses it to stop a model drifting far from where it started.

Why This Is the Training Loss

For one training example, the "true distribution" over the next token is one-hot: probability 1 on the token $c$ that actually came next. The cross-entropy collapses to one term:

H(\text{one-hot}_c, q) = -\log q(c)

That is the NLL of section 3.4. Minimizing cross-entropy, minimizing NLL and maximizing likelihood are the same thing. This cross-entropy loss trains every classifier and language model in this course.

Perplexity

is $\exp$ of the average NLL per token. A perplexity of 20 means the model is as unsure as if choosing uniformly among 20 tokens; 1 is perfect. Section 13.1 explains why perplexities across different tokenizers cannot be compared.

Figure 3.5

Cross-entropy never beats entropy; the gap is the KL divergence

The left bars are the true distribution p. Drag the right bars to change the model q; they rescale so they always add to 1.

All quantities in bits (log base 2).

Predict, then try

Drag the model's distribution $q$ until it matches the true distribution $p$ exactly. What will the cross-entropy and the KL divergence read then?

In words. Two bar charts over four outcomes: a fixed true distribution and a draggable model distribution. The readout shows entropy, cross-entropy and KL divergence. Cross-entropy never drops below entropy, and the gap, the KL divergence, closes only when the bars match.

3.6 The Sigmoid and Preferences Between PairsRun this section

The squashes any real score into a probability between 0 and 1:

\sigma(z) = \frac{1}{1 + e^{-z}}

$\sigma(0) = 0.5$ , and $\sigma(-z) = 1 - \sigma(z)$ . Its slope is $\sigma(z)(1 - \sigma(z))$ , the " $p(1-p)$ " from Module 2's exercises, because the sigmoid is the two-class softmax: $\frac{e^a}{e^a + e^b} = \sigma(a - b)$ .

That makes it the natural model for comparisons. Give each item a score $r$ , and say item $i$ beats item $j$ with probability

P(i \succ j) = \sigma(r_i - r_j)

Only the difference matters, so adding a constant to every score changes nothing. This is the Bradley–Terry model, long used to rate chess players from match results by maximum likelihood. Section 21.4 fits it to people's judgments of which of two answers is better, and calls the scores .

Exercises

Check

Which of these need the two variables to be independent? (a) $\mathbb{E}[X + Y] = \mathbb{E}[X] + \mathbb{E}[Y]$ ; (b) $\mathbb{E}[XY] = \mathbb{E}[X]\mathbb{E}[Y]$ ; (c) $\operatorname{Var}(X + Y) = \operatorname{Var}(X) + \operatorname{Var}(Y)$ .

Apply

Prove that $\operatorname{Var}(aX) = a^2 \operatorname{Var}(X)$ from the definition $\operatorname{Var}(X) = \mathbb{E}[X^2] - (\mathbb{E}[X])^2$ .

Apply

$X$ is a sum of 256 independent terms, each with mean 0 and variance 1. What is its standard deviation?

Apply

Factor $P(a, b, c, d)$ with the chain rule of probability, in left-to-right order.

Apply

A model gives probability $\tfrac{1}{20}$ to every token of a test text. What is its perplexity?

Apply

What is the entropy, in bits, of a distribution that gives probability $\tfrac{1}{2}, \tfrac{1}{4}, \tfrac{1}{4}$ to three outcomes?

Apply

Show that $\operatorname{KL}(p \| p) = 0$ , and compute $\sigma(2) + \sigma(-2)$ .

Extend

Under the Bradley–Terry model, answer A has reward 1.5 and answer B has reward 0.5. What is the predicted probability that A is preferred, and what would it be if both rewards were raised by 10?

Core Academic References

Cover & Thomas (2006): Elements of Information Theory (2nd ed.). Wiley.
Shannon (1948): A Mathematical Theory of Communication. Bell System Technical Journal, 27: 379–423.
Bradley & Terry (1952): Rank Analysis of Incomplete Block Designs: I. The Method of Paired Comparisons. Biometrika, 39(3/4): 324–345.
Goodfellow, Bengio & Courville (2016): Deep Learning, chapter 3 (Probability and Information Theory). MIT Press.