Glossary

$V$ is the vocabulary: the fixed list of tokens the tokenizer can produce. $|V|$ counts them. It is 50,257 for GPT-2, 32,000 for LLaMA-2 and 256,000 for Gemma.

Do not confuse it with two similar notations. $\|\mathbf{v}\|$ (double bars) is the length of a vector. $|x|$ on a single number is its absolute value, so $|-3| = 3$.

Each step runs a batch through the model (the forward pass), computes the loss, works out how the loss changes with each weight (backpropagation, section 2.5), and moves every weight a small step downhill. The step size is the learning rate $\eta$.

Using a trained model without changing it is called inference. Sections 10.2 and 21.5 cover how large models are trained.

Two facts explain how the course uses it:

• $e^{z} > 0$ for every real $z$. So softmax can never output a negative probability.
• $e^{a+b} = e^a e^b$. So subtracting the same constant from every input of a softmax multiplies the top and bottom of the fraction by the same factor, and it cancels exactly.

As $z$ goes to $-\infty$, $e^{z}$ goes to $0$. That is how the causal mask works: a score of $-\infty$ becomes a weight of exactly zero.

$\log(ab) = \log a + \log b$. A sentence's probability is a product of hundreds of small numbers. Multiplied directly in floating point, the result gets so small it rounds to zero (called underflow). Taking the log turns the product into a sum of moderate negative numbers, which computers handle easily.

$\ln$ is the natural log, the log to base $e$. In machine learning, $\log$ almost always means $\ln$ as well. A probability is at most 1, so its log is zero or negative. That is why losses put a minus sign in front: it makes the loss a positive number to minimize.

$\sum_{i=1}^{d} x_i$ means: set $i$ to 1 and take $x_1$, set $i$ to 2 and take $x_2$, and so on up to $x_d$, then add them all.

In Python it is sum(x[i] for i in range(1, d + 1)). When there is a set or condition below instead of a start value, such as $\sum_{i \in S}$, it means "add one term for every member of $S$". A bare $\sum_i$ means "over every value of $i$".

Which of the two depends on the letter's font. $x_i$ (plain) is one number inside a vector; $\mathbf{x}_i$ (bold) is the whole vector for token $i$. Two subscripts pick an entry of a matrix: $M_{ij}$ is the entry in row $i$, column $j$.

By habit, $i$ and $j$ count features or positions, $t$ counts time steps, $l$ counts layers, and $m$ and $n$ are positions in RoPE. Some subscripts are names rather than counters, such as the "model" in $d_{\text{model}}$.

It is the one size that stays the same through every layer. Every vector in the residual stream, and every sub-layer's input and output, has $d_{\text{model}}$ numbers. That is what lets the residual addition $\mathbf{x} + \operatorname{SubLayer}(\mathbf{x})$ work at every layer: you can only add two vectors of the same length.

The weight matrices in each layer are roughly $d_{\text{model}} \times d_{\text{model}}$ in size, so a layer's parameter count grows with $d_{\text{model}}^2$. Doubling the width roughly quadruples it.

$\mathcal{L}(\theta)$ is a function of the model's parameters $\theta$: plug in the current weights and get back a score for how wrong the model is on the training data. Training runs gradient descent on this number, repeatedly nudging the weights in the direction that lowers it.

A subscript names which loss: $\mathcal{L}_{\text{NLL}}$ (negative log-likelihood), $\mathcal{L}_{\text{CTC}}$ (a loss for speech alignment), $\mathcal{L}_{\text{aux}}$ (the extra balancing term in mixture-of-experts). A model's total loss is often a weighted sum of several of these.

A matrix has two axes, so two indices pick one entry: $X_{ij}$ is row $i$, column $j$. Its shape is always said rows first: $\mathbb{R}^{3 \times 4}$ is 3 rows by 4 columns, 12 numbers in all.

Matrices play two roles in a transformer:

• Data. A sentence of $N$ tokens, each a vector of width $d_{\text{model}}$, is stacked into an $N \times d_{\text{model}}$ matrix, one token per row.
• Weights. A learned $W$ turns vectors of one width into another by matrix multiplication.

In PyTorch a 3×4 matrix has shape (3, 4).

The raised $d$ is the length of the list, not a power. $\mathbb{R}^{768}$ is every possible vector of 768 numbers. Writing $\mathbf{x} \in \mathbb{R}^{768}$ tells you the shape of $\mathbf{x}$ (768 numbers) and nothing about what those numbers are.

When the raised part has two sizes joined by $\times$, it describes a matrix: $\mathbb{R}^{m \times n}$ is every table with $m$ rows and $n$ columns. The course states every tensor shape this way.

The double-struck $\mathbb{R}$ ("blackboard bold") is the set of real numbers, such as $-3$, $0$, $\tfrac{1}{7}$, $\pi$ and $2.718\ldots$. It does not include complex numbers. Every weight and activation in a model is a real number; a computer's floating-point numbers are a finite approximation of them.

Related sets you will meet: $\mathbb{N}$ (counting numbers $0, 1, 2, \dots$ or $1, 2, 3, \dots$ depending on the author), $\mathbb{Z}$ (all integers, including negatives), and $\{0,1\}$ (only zero and one, as in a one-hot vector).

$a \in S$ says that $a$ is one of the things in the set $S$. In this course it mostly does one job: stating a shape. $\mathbf{q} \in \mathbb{R}^{64}$ means "$\mathbf{q}$ is a vector of 64 real numbers", like a type annotation in code.

It is also used with a written-out set: $i \in \{0, 1, \dots, |V|-1\}$ says $i$ is one of the whole numbers in that range.

An axis is one direction you can step along: along a vector, down the rows of a matrix, across its columns. The shape records how many steps each axis has. The course writes a shape in the exponent of $\mathbb{R}$, as in $\mathbb{R}^{2 \times 3 \times 4}$, and PyTorch prints it as torch.Size([2, 3, 4]).

Most operations have a shape rule. Adding two tensors needs the same shape and keeps it. Multiplying an $m \times n$ matrix by an $n \times p$ one needs the inner sizes to agree and gives $m \times p$. When a formula confuses you, write the shape under each term; a mismatch usually shows where you misread it.

A scalar has no axes: there is nothing to index, because there is only one value. Writing $x \in \mathbb{R}$ says $x$ is one real number.

In PyTorch a scalar is a tensor of shape (), an empty shape. Multiplying a vector or matrix by a scalar scales every entry by the same amount.

A vector has one axis, and its length is how many numbers it holds. $\mathbf{x} \in \mathbb{R}^{4}$ is a vector of 4 numbers, and $x_2$ is its second entry.

The course writes vectors in bold and, following most transformer code, as rows: a token vector sits on the left of a matrix, as in $\mathbf{x} W$. Geometrically a vector is an arrow from the origin, which is what gives it a length and a direction (section 1.5).

In PyTorch a vector of 4 numbers has shape (4,).

The font tells you what kind of object a letter is:

• $x$: a scalar, which means one number
• $\mathbf{x}$: a vector, a list of numbers
• $X$ or $W$: a matrix, a 2-D table of numbers

This matters for subscripts. $\mathbf{x}_i$ is the whole vector for token $i$. $x_i$ is the $i$-th number inside a vector. Only the boldness tells the two apart.

Read a matrix by its shape first: count the rows, then the columns, and say "rows by columns". A matrix with 4 rows and 2 columns is "4 by 2", written $4 \times 2$. As a weight matrix acting on a row vector, a $4 \times 2$ matrix turns a 4-number input into a 2-number output. Often the shape is all you need to follow the argument.

(In the LaTeX source, & separates columns and \\ starts a new row.)

One vector would have to serve as a search term, a search label and the content itself, and those roles would get in each other's way. So the model learns three separate weight matrices and makes three versions of each token:

• $\mathbf{q}_i = \mathbf{x}_i W^Q$: what am I looking for?
• $\mathbf{k}_i = \mathbf{x}_i W^K$: what do I offer?
• $\mathbf{v}_i = \mathbf{x}_i W^V$: what do I contribute if chosen?

Attention compares every query with every key to get scores, then mixes the values using those scores. The raised $Q$, $K$, $V$ are labels, not powers. A fourth matrix, $W^O$, maps the mixed result back to the model's width.

Entry $(i,j)$ of $AB$ is the dot product of row $i$ of $A$ with column $j$ of $B$. For that to work, a row of $A$ must be as long as a column of $B$, so the inner sizes must match. Two things to remember:

• Shapes tell you what happens. $\mathbf{x}W^Q$, with $\mathbf{x}$ of width $d_{\text{model}}$ and $W^Q$ of shape $d_{\text{model}} \times d_k$, produces a query of width $d_k$. The shared middle size disappears.
• Order matters. $AB$ and $BA$ are usually different, and one of them may not even be defined.

Every "projection" in the course is one matrix multiplication. In PyTorch it is A @ B.

Hardware is fastest when it does many identical computations at once, so models process several sequences together. A batch of $B$ sentences, each $N$ tokens of width $d_{\text{model}}$, is one tensor of shape $B \times N \times d_{\text{model}}$.

The examples in a batch never mix, which is why most formulas in the course leave the batch axis out and describe one sequence. In training, each gradient step averages the loss over one batch (section 2.5).

Each extra axis is one more index. A stack of matrices is a 3-axis tensor: a batch of $B$ sentences, each $N$ tokens by $d_{\text{model}}$ numbers, has shape $B \times N \times d_{\text{model}}$. A color image is another: height by width by 3 color channels (section 25.2).

Multiply the sizes to count the numbers: a $2 \times 3 \times 4$ tensor holds 24.

Two meanings of rank. Some texts call the number of axes a tensor's rank or order. Section 20.2 uses rank for something else, the number of independent directions in a matrix. This course says "number of axes" to keep them apart; PyTorch calls it ndim.

$(\mathbf{a} \odot \mathbf{b})_i = a_i b_i$. The output has the same shape as the inputs. In NumPy and PyTorch it is a * b, while matrix multiplication is a @ b. It is also called the Hadamard product.

It appears wherever each feature gets its own scale: the learned gain $\boldsymbol{\gamma}$ in LayerNorm and RMSNorm, and the gate in SwiGLU, where one branch scales the other entry by entry. Some texts write $\otimes$ for the same thing, though $\otimes$ can also mean other kinds of product.

With $d_{\text{model}} = 512$ and $h = 8$ heads, $d_k = d_v = 512 / 8 = 64$. Splitting the width this way means multi-head attention costs about the same as one full-width head, while each head can learn to focus on a different kind of relationship.

$d_k$ is the width of queries and keys, which must match because they are dotted together. $d_v$ is the width of the values. They are usually equal but do not have to be. $d_{\text{head}}$ is another name for the same per-head width.

Entry by entry, $(A^\top)_{ij} = A_{ji}$: the number in row $i$, column $j$ of $A^\top$ is the number in row $j$, column $i$ of $A$. A $3 \times 5$ matrix becomes $5 \times 3$.

In this course its main job is making shapes fit so a multiplication is allowed. $QK^\top$ works because $Q$ is $N \times d_k$ and $K^\top$ is $d_k \times M$, giving the $N \times M$ matrix of scores: one row per query, one column per key. In self-attention $M = N$, so it is square. $QK$ without the transpose would not be a valid product.

$\mathbf{u}^\top \mathbf{v}$ is the dot product written as a matrix product: a $1 \times d$ row times a $d \times 1$ column gives one number.

A $4096 \times 4096$ matrix stores about 16.8 million numbers, but its rank can be as low as 8: every output it produces is then a mix of just 8 directions. Low rank means the matrix can be stored much more compactly, as two thin matrices.

LoRA assumes that the change a fine-tune makes to the weights has low rank, even though the weights themselves do not. So the change can be learned as two thin matrices.

$\langle \mathbf{u}, \mathbf{v}\rangle = \sum_i u_i v_i$. It also equals $\|\mathbf{u}\|\,\|\mathbf{v}\|\cos\theta$, where $\theta$ is the angle between the vectors. The second form is why attention uses it: the dot product is large and positive when two vectors point the same way, near zero when they are perpendicular, and negative when they point apart.

Angle brackets $\langle \mathbf{u}, \mathbf{v}\rangle$, $\mathbf{u}^\top\mathbf{v}$ and $\mathbf{u}\cdot\mathbf{v}$ are three ways of writing the same operation.

$\|\mathbf{x}\|_2 = \sqrt{\sum_i x_i^2}$. This is Pythagoras' theorem with $d$ sides instead of two. The small subscript says which kind of length:

• $\|\cdot\|_2$: ordinary (Euclidean) length, the default
• $\|\cdot\|_F$: the Frobenius norm, the same formula applied to every entry of a matrix
• $\|\cdot\|_1$: the sum of the absolute values of the entries

Dividing a vector by its norm gives a vector of length 1 (a unit vector). Cosine similarity does this to both vectors before taking their dot product.

$P_\theta(w_t \mid \cdot)$ means "the probability that this model, with its current weights $\theta$, gives to $w_t$". The subscript is a reminder that the probabilities change as training changes the weights.

$\theta$ has several meanings in this course. In Module 1 it is the angle between two vectors. In RoPE (Module 15) it is a rotation frequency. In the repetition penalty (Module 8) it is a single number you choose, such as 1.15. Read the surrounding sentence to know which.

A tokenizer splits text into tokens and looks each one up in the vocabulary, giving a whole number. The model then replaces each number with a vector from its embedding matrix (section 1.5). Common words are usually one token; rare words are split into several (section 9.2).

A sentence of $N$ tokens therefore becomes $N$ vectors, stacked as the rows of an $N \times d_{\text{model}}$ matrix.

$\mathbf{e}_i \in \{0,1\}^{|V|}$ has $|V|$ entries and a single 1, at position $i$. Multiplying it by a matrix picks out row $i$ of that matrix: $\mathbf{e}_i^\top W_E = W_E[i,:]$.

No real implementation does that multiplication; it just reads row $i$ from memory (nn.Embedding in PyTorch). Writing it as a multiplication shows that the embedding step is an ordinary matrix product like every other layer. It also makes the gradient easy to see: only row $i$ of $W_E$ gets updated.

$W_E \in \mathbb{R}^{|V| \times d_{\text{model}}}$: $|V|$ rows, each $d_{\text{model}}$ numbers wide. Row $i$ is the vector the model uses for token $i$. It is trained like any other weight.

Many models tie this matrix to the output layer: they reuse it (transposed) to turn the final hidden vector back into one score per vocabulary token. That saves a whole $|V| \times d_{\text{model}}$ block of parameters, which is a large share of a small model.

• $\mathcal{O}(1)$: constant, the same at any size.
• $\mathcal{O}(N)$: linear, double the input and double the work.
• $\mathcal{O}(N^2)$: quadratic, like building the $N \times N$ attention matrix.
• $\mathcal{O}(N^3)$: cubic, like generating $N$ tokens while recomputing all attention from scratch each step, with no KV cache.

Constant factors are dropped on purpose, so big-O answers "how does this scale?" and not "how fast is it?". At real sizes, a well-optimized $\mathcal{O}(N^2)$ method can beat a poorly written $\mathcal{O}(N\log N)$ one.

Everything else a model computes is derived from its input and its weights. The weights are fixed while the model runs and change only during training. The words weights and parameters mean the same thing; $\theta$ stands for all of them at once.

Memory is weights times bytes per number: 124 million weights stored as 16-bit numbers take about 248 MB.

A computer cannot store an arbitrary real number, so each entry is rounded to a fixed format. Floating-point formats keep a sign, an exponent (the range) and a mantissa (the precision): fp32 is standard, while bf16 and fp16 halve the memory. Integer formats such as int8 store whole numbers plus a shared scale, the subject of section 24.2.

The memory of a tensor is its number of entries times the bytes per entry. That is why the same model can need 4, 2 or 1 byte per weight depending on its data type.

Weights are what the model knows; activations are what it is thinking about this input. They are tensors too, usually of shape $B \times N \times d_{\text{model}}$ between layers. Training keeps them in memory so backpropagation can use them, which is why a long sequence costs memory even when the model is small. Section 24.2 shows why activations are harder to quantize than weights.

Do not confuse this with an activation function such as GELU, the nonlinearity applied inside the feed-forward network.

$\lim_{T \to 0^+} p_i$ asks what the softmax probabilities approach as the temperature $T$ shrinks toward zero from above (the $^+$ means "from the positive side"). The answer is that all probability goes to the largest logit. This describes greedy decoding, even though setting $T = 0$ directly would mean dividing by zero.

Limits let a formula describe its own edge cases.

$p_j \to 0$ means "$p_j$ approaches zero". $\theta_{\text{base}} = 10{,}000 \longrightarrow 500{,}000$ means the constant was changed from 10,000 to 500,000. $z_i \longleftarrow \dots$ is an assignment: replace $z_i$ with the right-hand side, like z[i] = ... in code.

An arrow with a condition written under it, such as $\xrightarrow[T-t\to\infty]{}$, says what the quantity approaches and what change drives it there.

$\approx$ usually marks a deliberate approximation, such as the tanh formula for GELU or a rounded parameter count. $\hat{x} \approx x$ after quantization admits that the scheme introduces some error.

$X \sim \mathcal{N}(0,1)$ reads "$X$ is drawn from a normal distribution with mean 0 and variance 1". $y \propto x$ means $y = c\,x$ for some fixed constant $c$.

$\Delta W$ is one object, the update to $W$, not $\Delta$ multiplied by $W$. In LoRA, the fine-tuned weights are $W_0 + \Delta W$, where $\Delta W$ is the learned correction.

$\frac{\partial \mathcal{L}}{\partial \theta}$ is the slope of the loss with respect to one parameter. Its sign tells training which way to move that parameter, and its size tells how much the loss cares. The curly $\partial$ (instead of the $d$ in $\frac{dy}{dx}$) signals that the function has several inputs and all the others are held still.

When the top and bottom are both vectors, the result is a matrix holding the slope of every output with respect to every input. That matrix is called a Jacobian.

$\nabla_\theta \mathcal{L}$ collects $\partial\mathcal{L}/\partial\theta_k$ for every parameter $\theta_k$ into one vector, the same shape as the parameters. Training takes a small step in the opposite direction, $\theta \leftarrow \theta - \eta\,\nabla_\theta \mathcal{L}$ with a small step size $\eta$, because the fastest way uphill, reversed, is the fastest way down.

In $\theta \leftarrow \theta - \eta \, \nabla_\theta \mathcal{L}$, the gradient says which way to move and $\eta$ says how far. Too small and training crawls; too large and every step overshoots. Large models do not hold it fixed: section 10.2 raises it during a warmup and then lowers it along a schedule.

$\sigma = \sqrt{\operatorname{Var}}$. Taking the square root puts the spread back in the same units as the data, so you can compare it with the values themselves. $\sigma^2$ is the variance.

In Section 20.2 the same letter is used for singular values $\sigma_1 \ge \sigma_2 \ge \dots$, which say how strongly a matrix stretches along each of its main directions. The two meanings are unrelated; a numbered subscript ($\sigma_1$, $\sigma_i$) usually means a singular value.

Written as a function, $\sigma(z) = \frac{1}{1 + e^{-z}}$ is the sigmoid, which squashes any number into the range 0 to 1. That is its meaning in Swish, $z \cdot \sigma(\beta z)$, in section 15.4: the sigmoid acts as a soft on/off switch on $z$.

In LayerNorm, $\mu = \frac{1}{d}\sum_i x_i$ is the average of the $d$ numbers in one token's vector, not an average across the tokens in a batch. Subtracting it shifts that token's values so they average to zero.

RMSNorm skips this subtraction, on the argument that it costs compute without helping much.

In this course it mostly undoes a squaring. Variance is measured in squared units, so $\sqrt{\operatorname{Var}}$ brings the spread back to the scale of the data. That is why attention divides by $\sqrt{d_k}$ and not by $d_k$: the score's variance is $d_k$, so its typical size is $\sqrt{d_k}$.

$\mathbb{E}[X]$ is what $X$ averages out to over a very large number of draws. Two rules are used in the course:

• It is linear, always: $\mathbb{E}[aX + bY] = a\mathbb{E}[X] + b\mathbb{E}[Y]$.
• For independent quantities (knowing one tells you nothing about the other), the expected value of a product splits: $\mathbb{E}[XY] = \mathbb{E}[X]\,\mathbb{E}[Y]$.

The second rule only holds under independence, and the proof behind the $\sqrt{d_k}$ scaling depends on it.

$\operatorname{Var}(X) = \mathbb{E}[X^2] - (\mathbb{E}[X])^2$: the average squared distance from the mean. Two rules are used repeatedly in the course:

• For independent quantities, variances add: $\operatorname{Var}(X + Y) = \operatorname{Var}(X) + \operatorname{Var}(Y)$.
• Scaling by a number $c$ scales the variance by $c^2$: $\operatorname{Var}(cX) = c^2 \operatorname{Var}(X)$.

If the entries of $\mathbf{q}$ and $\mathbf{k}$ are independent with mean 0 and variance 1, each product $q_i k_i$ has variance 1, so the first rule gives $\operatorname{Var}(\mathbf{q}^\top\mathbf{k}) = d_k$. The second rule then shows that dividing by $\sqrt{d_k}$ brings it back to 1.

$P(w_t \mid w_1, \dots, w_{t-1})$ is the whole job of a language model: the probability of each possible next token, given all the tokens so far. It is a full list of probabilities, one for every token in the vocabulary, not a single word. Sampling picks one token from that list.

$w_{<t}$ is shorthand for the same list of earlier tokens, $w_1, \dots, w_{t-1}$.

This is exact, not an approximation. It is the chain rule of probability applied from left to right. Its consequence shapes the whole architecture: a model that can predict one next token can score or generate text of any length by repeating that step. A model that works this way is called autoregressive.

For a coin, the maximum-likelihood estimate is the observed frequency of heads (section 3.4).

Measured in bits with log base 2 (section 3.5). It is the floor that cross-entropy can never go below.

Minimizing it is the same as maximizing likelihood (sections 3.4 and 3.5). It is the loss of every classifier and language model in the course.

$\operatorname{KL}(P \,\|\, Q) = \sum_x P(x) \log \frac{P(x)}{Q(x)}$. In RLHF (section 21.5) it measures how far the trained model has drifted from the model it started from, and the objective subtracts it as a penalty so the model stays fluent while it chases reward.

Lower is better. It is per token, so it cannot be compared across tokenizers; bits per byte can (section 13.1).

It is the two-class softmax, and the Bradley–Terry model of preferences uses it on a difference of scores (section 3.6).

Section 21.4 trains a reward model from pairwise human preferences; section 22.3 uses verifiable rewards.

A language model is trained on next-token prediction: hide the next token, predict it from the ones before. Every position of every text is a labelled example for free, which is why the training signal grows with the supply of text (section 4.1). Masked-token prediction (BERT) and autoencoders are other self-supervised tasks.

A layer is many neurons reading the same input, which is one matrix multiplication followed by the activation function applied entry by entry (section 4.2).

Without it, stacked linear layers collapse into a single linear layer, so depth would add nothing (section 4.2).

Hidden layers sit between input and output; their size is the network's width (section 4.2).

The last layer of a language model outputs a vector $\mathbf{z}$ with one entry per token in the vocabulary. A larger logit means that token is more likely to come next, but logits are not probabilities: they do not add to 1 and can be negative. Softmax converts them (section 4.3), and temperature divides them first to sharpen or flatten the result.

$\operatorname{Softmax}(\mathbf{z})_i = e^{z_i} / \sum_j e^{z_j}$. Taking $e$ to the power of each input makes every value positive. Dividing by the total makes them add up to 1.

It is called soft because it does not just pick the largest input (that would be a "hard" max). Larger inputs get a larger share, but every entry keeps some probability above zero. Only the differences between inputs matter: adding the same constant to every input leaves the output unchanged.

The attention mask uses $-\infty$ because adding it to any score makes the result $-\infty$, and $e^{-\infty} = 0$: the position gets zero weight. Code uses a large finite negative number instead, because true infinities cause NaN ("not a number") in some arithmetic, such as $\infty - \infty$ or $0 \times \infty$. For example, a row where every position is masked with $-\infty$ would give NaN.

Dividing by $T < 1$ makes the gaps between logits bigger, so softmax puts more probability on the top tokens: safer, more repetitive text. $T > 1$ makes the gaps smaller, pushing toward equal probabilities: more varied text, and eventually nonsense.

The division happens before the exponential. So the ratio between any two probabilities, $p_i / p_j = e^{(z_i - z_j)/T}$, gets raised to the power $1/T$. The effect is a power on probability ratios, not a simple shift of the probabilities.

$\max_k z_k$ is the biggest logit. $\arg\max_k z_k$ is which token has it. Greedy decoding outputs the $\arg\max$ token at every step.

The difference shows up in the safe softmax too: there you subtract $\max_k z_k$, the largest value, from every logit to prevent overflow.

Each row has a value, then a condition. Exactly one condition should be true for any input. It is if / elif / else in maths notation. The causal mask, the repetition penalty and the multimodal label mask are written this way.

$\delta_{ij} = 1$ if $i = j$, and $0$ otherwise. It lets one formula cover two cases. In the softmax Jacobian, $p_i(\delta_{ij} - p_j)$ becomes $p_i(1 - p_i)$ when $i = j$ and $-p_i p_j$ when $i \ne j$.

Written as a table over all $i$ and $j$, it is exactly the identity matrix.

Suppose the entries of $\mathbf{q}$ and $\mathbf{k}$ are independent, with mean 0 and variance 1. Their dot product adds $d_k$ such products, so its variance is $d_k$ and its standard deviation is $\sqrt{d_k}$. At $d_k = 64$ that is 8. Scores that far apart push softmax to its extremes: one weight near 1 and the rest near 0. There its slope, of the form $p(1-p)$, is close to zero, so learning stalls.

Dividing by $\sqrt{d_k}$ brings the variance back to 1, keeping the weights spread out and the gradients useful.

Measured on a held-out validation or test set (section 4.4). A model that memorizes its training examples fits them perfectly and generalizes badly.

Chosen by comparing validation loss across candidate values (section 4.5).

The chain rule says the derivative through a sequence of functions is the product of each function's derivative. Across $T - t$ recurrent steps, that is $T - t$ matrices multiplied together.

If each factor shrinks things (roughly, a size $\lambda < 1$), the product shrinks like $\lambda^{T-t}$. The gradient reaching step $t$ is then effectively zero, and the model cannot learn that step $t$ affected step $T$. If each factor is above 1, the product grows without limit instead. Keeping every factor near exactly 1 over long sequences is very hard. This is a main reason attention replaced recurrence: attention connects any two positions in a single step.

$\prod_{t=1}^{T} a_t = a_1 \times a_2 \times \cdots \times a_T$.

It appears in two places in the course. First, chaining probabilities: a sentence's probability is the product of each token's probability given the tokens before it. Second, chaining derivatives through many steps. This is why RNN gradients vanish: multiply $T$ numbers that are each below 1 and the result shrinks toward zero, faster the larger $T$ is.

None of these meanings carries over to the next. In LoRA, $\frac{\alpha}{r}$ scales the low-rank update. In mixture-of-experts training, $\alpha$ (often 0.01) sets how much the balancing loss counts. In SmoothQuant, $\alpha$ (usually 0.5) splits an outlier's size between activations and weights. In CTC, $\alpha_t(s)$ is a running total of path probabilities. The sentence around the symbol always says which one is meant.

$\operatorname{Attention}(Q,K,V) = \operatorname{Softmax}\!\left(\frac{QK^\top}{\sqrt{d_k}} + M\right)V$

Read it step by step, with $N$ tokens:

1. $QK^\top$ is an $N \times N$ table: how well each token's query matches each token's key. (In cross-attention the keys come from a different sequence of length $M$, and the table is $N \times M$.)
2. Divide by $\sqrt{d_k}$ so the scores stay in a range where softmax still responds to differences.
3. Add the mask $M$ to block positions a token must not see.
4. Softmax each row, so each token's weights add up to 1.
5. Multiply by $V$: each token gets a weighted blend of the value vectors.

Adding $-\infty$ to a score before softmax removes that option. Since $e^{-\infty} = 0$, that position gets exactly zero weight, contributes nothing, and receives no gradient.

The causal mask sets $M_{ij} = -\infty$ whenever $j > i$, so token $i$ cannot look at later tokens. In code, a large negative number such as $-10^9$ is used in place of $-\infty$ to avoid arithmetic with true infinities.

$[\operatorname{head}_1; \dots; \operatorname{head}_h]$ places $h$ vectors, each $d_v$ wide, side by side into one vector $h\,d_v$ wide. That is why $W^O$ has $h\,d_v$ rows: one for each number in the joined vector.

When the items are matrices whose rows are tokens, the semicolon stacks their rows: $[\mathbf{X}_{\text{img}}; \mathbf{X}_{\text{text}}]$ is the image rows followed by the text rows, one sequence. The same notation adds a new row to the KV cache and inserts image patches among text tokens in a multimodal sequence. In PyTorch it is torch.cat.

The course uses $N$ in the attention chapters, $T$ for time steps in generation, and $L$ for context length in the KV-cache calculations. All three mean the same thing; the different letters come from the original papers.

It is the number that appears squared in attention's cost, so many of the optimizations in Module 15 aim to reduce what it costs.

Watch the letters. In section 8.5, $L$ counts layers, not tokens. When a formula has both, read the sentence around it to see which is which.

Because each block adds to the stream instead of overwriting it, both information and gradients have a direct path from the first layer to the last.

Where the normalization sits is the Pre-LN versus Post-LN question. Normalizing inside the branch, before the sub-layer, leaves the direct path untouched. Normalizing after the addition puts a normalization on that path at every layer, and deep models built that way usually need a learning-rate warmup to train.

In the Post-LN form, $\mathbf{x}_{l+1} = \operatorname{LayerNorm}(\mathbf{x}_l + \operatorname{Sublayer}(\mathbf{x}_l))$, the residual addition is still there; it is just normalized afterwards.

These are unrelated uses that follow each paper's own notation. With a subscript number, $\beta_1$ and $\beta_2$ are Adam's; bold $\boldsymbol{\beta}$ is LayerNorm's shift vector; with a time subscript, $\beta_t$ is the delta rule's write strength. In RLHF and DPO, $\beta$ sets how far the trained model may move from the reference model.

$\operatorname{LayerNorm}(\mathbf{x}) = \frac{\mathbf{x} - \mu}{\sqrt{\sigma^2 + \epsilon}} \odot \boldsymbol{\gamma} + \boldsymbol{\beta}$. Here $\mu$ and $\sigma^2$ are the mean and variance of the numbers within one token's vector. So the result for one token does not depend on the other examples in the batch, unlike BatchNorm. $\boldsymbol{\gamma}$ (gain) and $\boldsymbol{\beta}$ (bias) are learned per feature.

It keeps activations in a steady range as layers stack up. Whether it is placed before the sub-layer (Pre-LN) or after the residual addition (Post-LN) affects whether deep models need a learning-rate warmup to train.

Typical values are $10^{-5}$ or $10^{-6}$. In $\sqrt{\sigma^2 + \epsilon}$ it is a safeguard: if all of a token's values happen to be equal, then $\sigma^2 = 0$ and dividing by $\sqrt{0}$ would give infinity or NaN ("not a number"). $\epsilon$ is small enough to have no visible effect normally and large enough to prevent that.