Shapes and Linear Algebra

A transformer is arithmetic on tables of numbers. This module gives you the toolkit to follow it: read a symbol such as $X \in \mathbb{R}^{N \times d_{\text{model}}}$ and know what it holds, multiply matrices by hand, and say what a multiplication keeps and what it throws away.

1.1 Scalars, Vectors, Matrices, and TensorsRun this section

Everything a transformer reads, stores and produces comes in one of four arrangements, which differ in their number of axes: directions you can step along from one number to the next. Know a symbol's arrangement and the length of each axis, and you can follow most formulas before you know what they compute.

Scalar: One Number

A scalar is a single number. It has no axes, because there is nothing to step along. The temperature that sets how random a model's output is (section 4.3), the step size used in training (section 2.4) and the loss that measures a model's error are all scalars. The course writes a scalar as a plain italic letter, and writes "$x$ is one real number" as $x \in \mathbb{R}$.

Vector: A List

A vector is a list of numbers in a fixed order. It has one axis, and its length is how many numbers it holds:

Vectors are written in bold. The raised $4$ is the length of the list, not a power. One index picks one entry: $x_3 = 3.0$ is the third number. The entry is written plain rather than bold because it is a scalar. Section 1.5 shows how every token of text becomes a vector like this one, only hundreds or thousands of numbers long.

Matrix: A Table

A matrix is a table of numbers with two axes, rows and columns. Take the three tokens of "the cat sat", give each a vector of four numbers, and stack the vectors as rows:

A matrix's size is said rows first, so this is a "3 by 4" matrix holding 12 numbers. Matrices are written as capital letters. Two indices pick one entry, row first: $X_{23} = -0.3$ is row 2, column 3. Row 2 on its own is the vector for "cat".

Some matrices hold data, like $X$, one row per token. Others hold learned numbers, like the $W^Q$ of section 1.6.

Tensor: A Stack of Tables

Add a third axis and you have a stack of matrices. A model processes several sentences at once, a batch, so the data entering a layer is a stack of matrices like $X$, one per sentence:

Here $B$ is the number of sentences in the batch, $N$ the number of tokens in each, and $d_{\text{model}}$ the length of each token's vector. An array with any number of axes is called a tensor. Two sentences of three tokens with four numbers each make a 2 by 3 by 4 tensor, and multiplying the sizes gives the 24 numbers it holds. A color image is a tensor too: height by width by three color channels (section 25.2).

In code, torch.Tensor holds all four arrangements (a scalar has zero axes). In prose, "tensor" usually means three axes or more.

In words. The figure draws one boxed number, a row of boxes, a grid, and a stack of grids like sheets of paper; the depth of the stack is the third axis. Clicking a box shows the indices that reach it (none, one, two or three), and the readout multiplies the axis lengths to count the numbers.

Reading a Shape

The list of axis lengths is the shape. The course writes it as the raised part of $\mathbb{R}$, and PyTorch prints it as a tuple from x.shape. Here are the four arrangements with sizes from GPT-2 Small, whose tokens are vectors of 768 numbers:

Sentences in a batch go through the same computation and never mix, so the course drops the batch axis and follows one sequence: attention in Module 6 works on an $N \times d_{\text{model}}$ matrix, and the code puts $B$ in front of every shape.

The course counts indices from 1, as mathematics does. PyTorch counts from 0, so $x_3$ is x[2] in code and $X_{23}$ is X[1, 2].

Some texts call the number of axes a tensor's "rank". This course reserves rank for something else (section 1.3) and says "number of axes".

1.2 Matrix Multiplication, Done by HandRun this section

Almost every line of a transformer is a matrix multiplication, so do one by hand until the rule is automatic.

Row $i$ of $A$ and column $j$ of $B$, multiplied entry by entry and summed, give entry $(i, j)$ of $AB$. With $n$ the shared length:

The $\sum$ ("sum") lets $k$ run from $1$ to $n$ and adds one term for each value. The row and column must pair up entry for entry, which is why the inner sizes have to agree.

Worked example. Multiply a 2 by 3 matrix by a 3 by 2 matrix: $A = \begin{bmatrix} 1 & 2 & 0 \\ -1 & 0 & 3 \end{bmatrix}$, $B = \begin{bmatrix} 2 & 1 \\ 0 & -1 \\ 1 & 4 \end{bmatrix}$. Entry $(1,1)$ is row 1 of $A$ with column 1 of $B$: $1 \cdot 2 + 2 \cdot 0 + 0 \cdot 1 = 2$. Entry $(1,2)$ is row 1 with column 2: $1 \cdot 1 + 2 \cdot (-1) + 0 \cdot 4 = -1$. Entry $(2,1)$ is $-1 \cdot 2 + 0 \cdot 0 + 3 \cdot 1 = 1$, and entry $(2,2)$ is $-1 \cdot 1 + 0 \cdot (-1) + 3 \cdot 4 = 11$. So $AB = \begin{bmatrix} 2 & -1 \\ 1 & 11 \end{bmatrix}$, a 2 by 2 matrix.

In words. The figure shows two small editable matrices and their product. Pointing at an entry of the product highlights the row and column that produce it and spells out the sum.

Two facts follow. Order matters: $AB$ and $BA$ usually differ, and often only one exists. And each row of $AB$ depends on one row of $A$ only, so multiplying a one-token-per-row matrix by $W$ transforms every token separately, with the same $W$.

What Operations Do to Shapes

Checking the shape of each output is the quickest way to follow a formula.

• Entry by entry. Adding two tensors, or multiplying them entry by entry ($\odot$), needs the same shape and keeps it.
• By a scalar. Multiplying by a scalar scales every entry and keeps the shape.
• Dot product. Two vectors of the same length give one scalar (section 1.4).
• Along an axis. Summing or averaging along an axis removes it: the average of each row of $X$ is a vector of 3 numbers. Softmax (section 4.3) keeps the shape but also works along one axis, turning each row into probabilities separately.
• Matrix multiplication. An $m$ by $n$ matrix times an $n$ by $p$ matrix needs the two inner sizes to agree, and gives an $m$ by $p$ matrix:

The shared $d_{\text{model}}$ disappears: every token's $d_{\text{model}}$ numbers become $d_k$ numbers. That is the shape of every projection in section 1.6.

Worked example. Multiply the 3 by 4 matrix $X$ above by a 4 by 2 matrix $W$. The inner sizes are both 4, so the product exists, and it is 3 by 2: three tokens, two numbers each. $W$ times $X$ is not defined at all, because a row of $W$ holds 2 numbers and a column of $X$ holds 3.

1.3 The Matrix ToolkitRun this section

These operations recur throughout the course; later sections use them without stopping.

Transpose

The transpose $A^\top$ swaps rows and columns: row $i$ of $A$ becomes column $i$ of $A^\top$, so a 3 by 2 matrix becomes 2 by 3, and $(A^\top)_{ij} = A_{ji}$. A row vector transposed is a column vector. One rule is used repeatedly: the transpose of a product reverses the order, $(AB)^\top = B^\top A^\top$.

Identity and Diagonal Matrices

The identity matrix $I$ is square, with 1 on its diagonal (top-left to bottom-right) and 0 elsewhere. Like the number 1, it changes nothing: $AI = IA = A$.

A diagonal matrix $\operatorname{diag}(\mathbf{s})$ has the entries of $\mathbf{s}$ on its diagonal and zeros elsewhere. On the right it scales each column of a matrix by one entry of $\mathbf{s}$; on the left, each row. $\operatorname{diag}(2, 3)$ doubles a vector's first entry and triples its second.

Inverse

The inverse $A^{-1}$ of a square matrix undoes it: $A^{-1}A = AA^{-1} = I$. A matrix that squashes some direction to zero has none, because that information is gone. A diagonal matrix with no zeros is easy to invert: divide instead of multiply, so $\operatorname{diag}(2, 3)^{-1} = \operatorname{diag}(\tfrac{1}{2}, \tfrac{1}{3})$. Sections 24.3 and 24.4 use this.

Element-Wise Product

The element-wise product $A \odot B$ multiplies entries in matching positions, so both matrices must have the same shape and the result does too. It is not matrix multiplication: $\begin{bmatrix} 1 & 2 \end{bmatrix} \odot \begin{bmatrix} 3 & 4 \end{bmatrix} = \begin{bmatrix} 3 & 8 \end{bmatrix}$. Some papers write it $\otimes$.

Outer Product

For column vectors $\mathbf{u}$ (length $m$) and $\mathbf{v}$ (length $n$), the outer product $\mathbf{u}\mathbf{v}^\top$ is the $m$ by $n$ matrix with entry $(i, j) = u_i v_j$. Every row is a multiple of $\mathbf{v}^\top$, so it is the simplest possible matrix. The reverse order, $\mathbf{u}^\top\mathbf{v}$, is 1 by 1: the dot product. Sections 18.1 and 20.2 build matrices from sums of outer products.

Independent Directions and Rank

Vectors are linearly independent when none can be made by scaling and adding the others: $[1, 0, 0]$ and $[0, 1, 0]$ are, but $[1, 1, 0]$ adds no new direction, being their sum. The rank of a matrix is the number of independent directions among its rows (equivalently, its columns), at most the smaller of its two sizes. Section 1.6 uses it to say how much of a token vector a projection keeps.

The Row-Vector Convention

This course writes a token's vector as a row (a 1 by $d_{\text{model}}$ matrix), so a weight matrix multiplies it from the right: $\mathbf{y} = \mathbf{x}W$. PyTorch's nn.Linear stores the matrix transposed and computes $\mathbf{x}W^\top$, the same thing. Sections 15.1 and 18.1 switch to column vectors, $\mathbf{y} = R\mathbf{x}$, to match their papers, and say so.

1.4 Dot Products, Length, and AngleRun this section

Geometric Meaning of the Dot Product

The model compares two vectors with the dot product (or inner product): multiply entry by entry and add. The same number is the product of the two lengths times the cosine of the angle between them.

$\|\mathbf{u}\|_2$ is the length of $\mathbf{u}$ (the square root of the sum of its squared entries) and $\theta$ the angle between the vectors, which makes sense in any number of dimensions.

In words. Vectors pointing the same way score high, perpendicular vectors score zero, and opposite vectors score negative. The angle sets the sign; the lengths set the size.

Perpendicular vectors are called orthogonal: neither has any component along the other.

Cosine Similarity vs. Unnormalized Dot Product

Divide out both lengths and only direction remains: cosine similarity, between $-1$ and $1$.

Attention uses the raw dot product, so a longer vector produces larger scores. Length becomes a second learned signal alongside direction, for example to make one token's scores sharper than another's. The cost is that raw dot products have no fixed range, which is why Module 2 scales them down.

1.5 Embeddings: From a Token to a VectorRun this section

A model cannot do arithmetic on text, so each token (a word, part of a word, or a punctuation mark) becomes a vector of fixed length:

The length $d_{\text{model}}$ is the model's width: $768$ in GPT-2 Small, $4096$ in LLaMA-3 8B and $12{,}288$ in GPT-3 175B, according to their published descriptions. Section 19.1 compares current models.

From Text to Vectors: The Embedding Lookup Walk-through

How "cat" becomes a vector of length $d_{\text{model}}$:

1. Tokenization. A fixed dictionary, the vocabulary $V$, gives every token a whole-number ID from $0$ to $|V|-1$, where $|V|$ is the number of entries. Say "cat" gets ID $3797$.
2. One-hot encoding. On paper, the ID can be written as a one-hot vector $\mathbf{e}_i$: a vector of length $|V|$ that is $1$ at position $i$ and $0$ everywhere else.
3. Row lookup. The embedding matrix $W_E$ has one row per vocabulary entry and $d_{\text{model}}$ columns ($|V| \times d_{\text{model}}$). The one-hot row $\mathbf{e}_i^\top$ times $W_E$ picks out row $i$, because every other row is multiplied by zero.
   $$\mathbf{x} = \mathbf{e}_i^\top W_E = W_E[i, :] \in \mathbb{R}^{d_{\text{model}}}$$

   What does this say?

Real code skips the multiplication and reads row $i$ straight from memory, which costs the same however large the vocabulary: $\mathcal{O}(1)$, read "order one", meaning the cost does not grow with input size. In PyTorch this is nn.Embedding(vocab_size, d_model). The rows start random and are learned in training, which nudges the model's numbers, its weights (or parameters), to make fewer mistakes on example text.

1.6 Linear Projections: One Vector Into ThreeRun this section

In attention, each token plays three roles: it looks for other tokens, it is found by them, and it hands information to those that find it. Rather than make one vector do all three jobs, the transformer makes three vectors from each token with three learned matrices:

$\mathbf{q}_i$ is the query, $\mathbf{k}_i$ the key and $\mathbf{v}_i$ the value of token $i$. Each matrix reads $d_{\text{model}}$ numbers and writes $d_k$ or $d_v$, usually far fewer. Multiplying by such a matrix is a linear projection.

In words. One token vector, three learned matrices: the query is what the token looks for, the key what it is matched on, and the value what it passes on.

Concrete Numerical Toy Example of Q, K, V Projections

Take $d_{\text{model}} = 4$, $d_k = d_v = 2$, two tokens $\mathbf{x}_1 = [1.0, \, 0.0, \, 2.0, \, -1.0]$ and $\mathbf{x}_2 = [0.0, \, 1.0, \, 1.0, \, 1.0]$, and these learned matrices:

Each output entry is the dot product of the input vector with one column of the matrix.

1. Query for token 1. The first column of $W^Q$ is $[1, 0, 1, 0]$ and the second is $[0, 1, -1, 1]$:
   $$\mathbf{q}_1 = \mathbf{x}_1 W^Q = [1(1) + 0(0) + 2(1) - 1(0), \;\; 1(0) + 0(1) + 2(-1) - 1(1)] = [3.0, \, -3.0]$$

   What does this say?
2. Key for token 2. The first column of $W^K$ is $[0, 1, -1, 1]$ and the second is $[1, 0, 1, 0]$:
   $$\mathbf{k}_2 = \mathbf{x}_2 W^K = [0(0) + 1(1) + 1(-1) + 1(1), \;\; 0(1) + 1(0) + 1(1) + 1(0)] = [1.0, \, 1.0]$$

   What does this say?
3. Raw score. Take the dot product of the query and the key:
   $$\mathbf{q}_1 \mathbf{k}_2^\top = (3.0 \times 1.0) + (-3.0 \times 1.0) = 3.0 - 3.0 = 0.0$$

   What does this say?

The raw embeddings have dot product $\mathbf{x}_1 \cdot \mathbf{x}_2 = 0 + 0 + 2 - 1 = 1$, but the query-key score is $0$: the score depends on the learned matrices. Training adjusts $W^Q$ and $W^K$ until tokens that should attend to each other score high.

Rank and Projection Subspaces

A $d_{\text{model}} \times d_k$ matrix has rank at most $d_k$ when $d_k$ is smaller (section 1.3), so each projection keeps only a $d_k$-dimensional slice of the token vector, and each matrix can learn a different slice:

• the query $\mathbf{q}_i$ holds the features token $i$ is searching for;
• the key $\mathbf{k}_j$ holds the features token $j$ can be found by;
• the score $\mathbf{q}_i \mathbf{k}_j^\top$ is high when the two sets of features line up.

Module 6 turns these scores into attention.

In words. The figure projects six two-dimensional token vectors onto a query line and a key line. Each projection keeps only the part along its line, so distant tokens can land close together. Turning a line changes which tokens end up near each other, as training does when it adjusts W Q and W K.

1.7 How Numbers Are StoredRun this section

Each entry of a tensor has a data type. The standard is 32-bit floating point (fp32), 4 bytes per number. Large models usually train in 16-bit formats (bf16 or fp16, 2 bytes) and are often served in 8 bits or fewer (Module 24). A tensor's memory is entries times bytes per entry. GPT-2's token embedding matrix has 50,257 rows and 768 columns, which is 38,597,376 numbers: about 154 MB in fp32 and 77 MB in bf16.

Two kinds of tensor fill memory: the learned matrices, fixed while the model runs, and the activations, computed fresh from each input.

Every symbol and term is collected in the glossary.

Exercises

Core Academic References

• Strang (2016): Introduction to Linear Algebra (5th ed.). Wellesley-Cambridge Press.
• Golub & Van Loan (2013): Matrix Computations (4th ed.). Johns Hopkins University Press.
• Mikolov et al. (2013): Efficient Estimation of Word Representations in Vector Space (word2vec). arXiv:1301.3781