\mathbf{h}

Activations

The numbers a model computes as it runs: every intermediate vector between the input and the output. Unlike weights, they change with every input.

Weights are what the model knows; activations are what it is thinking about this input. They are tensors too, usually of shape $B \times N \times d_{\text{model}}$ between layers. Training keeps them in memory so backpropagation can use them, which is why a long sequence costs memory even when the model is small. Section 24.2 shows why activations are harder to quantize than weights.

Do not confuse this with an activation function such as GELU, the nonlinearity applied inside the feed-forward network.

Example

X \in \mathbb{R}^{N \times d_{\text{model}}} \;\to\; \text{block}_1 \;\to\; \mathbf{h}^{(1)} \;\to\; \text{block}_2 \;\to\; \cdots

Each block reads activations and writes new ones of the same shape.

Where the course uses it

1.7 How Numbers Are Stored · Module 1, Shapes and Linear Algebrafirst use
11.1 Why One GPU Is Not Enough · Module 11, Training Across Many GPUs

Related terms

Weights (parameters)The numbers a model learns in training: the entries of its weight matrices and vectors. A model's size is counted in weights, such as 124 million for GPT-2 Small.
Tensor — numbers on any number of axesAn array of numbers with any number of axes. In code every array is called a tensor: a scalar has 0 axes, a vector 1, a matrix 2. In prose the word usually means 3 or more.
GELU, ReLU, Swish — activation functionsThe nonlinearity applied inside the feed-forward block. ReLU sets negative numbers to zero; GELU and Swish do almost the same but with a smooth curve.