I

I — the identity matrix

Read "I" or "the identity". It is a square matrix with 1s on the diagonal and 0s elsewhere. Multiplying by it leaves a vector unchanged.

In the residual stream, $\frac{\partial \mathbf{x}_L}{\partial \mathbf{x}_0} = I + \sum(\dots)$ . The $I$ comes from the skip connections. It means that even if every learned term contributed nothing, the gradient would still reach the early layers at full strength through the $I$ . This direct route is sometimes called the "gradient highway".

Example

\begin{bmatrix} 1 & 0 \\ 0 & 1 \end{bmatrix} \begin{bmatrix} 3 \\ 7 \end{bmatrix} = \begin{bmatrix} 3 \\ 7 \end{bmatrix}

The 2×2 identity returns its input.

Where the course uses it

8.5 Where the Norm Goes: Post-LN and Pre-LN · Module 8, Three Families: Encoder, Encoder-Decoder, Decoderfirst use

Related terms

δᵢⱼ — Kronecker deltaRead "delta i j". It equals 1 when i and j are the same number and 0 when they differ. It is an if-statement written as a number.
Residual connectionThe next vector is the current vector plus what the sub-layer computed. Each sub-layer adds a change instead of replacing its input. The subscript l counts sub-layers.