\mu

μ — the mean

Read "mu". It is the mean (the ordinary average) of a set of numbers: add them up and divide by how many there are.

In LayerNorm, $\mu = \frac{1}{d}\sum_i x_i$ is the average of the $d$ numbers in one token's vector, not an average across the tokens in a batch. Subtracting it shifts that token's values so they average to zero.

RMSNorm skips this subtraction, on the argument that it costs compute without helping much.

Example

\mathbf{x} = [1, 2, 6] \;\Longrightarrow\; \mu = \tfrac{1+2+6}{3} = 3

Subtracting the mean gives [-2, -1, 3], which averages to zero.

Where the course uses it

3.1 Random Variables, Distributions, and Sampling · Module 3, Probability and Informationfirst use
7.3 LayerNorm · Module 7, The Block and the Full Model

Related terms

LayerNorm — layer normalizationRead "layer norm". It shifts and rescales each token's vector to have mean 0 and variance 1, then applies a learned scale and shift.
σ — standard deviation, singular value, or sigmoidRead "sigma". Usually it is the standard deviation: a typical distance of values from their average. With a numbered subscript (section 20.2) it is a singular value, and written as a function, σ(z), it is the sigmoid.