Residual connection
The next vector is the current vector plus what the sub-layer computed. Each sub-layer adds a change instead of replacing its input. The subscript l counts sub-layers.
Because each block adds to the stream instead of overwriting it, both information and gradients have a direct path from the first layer to the last.
Where the normalization sits is the Pre-LN versus Post-LN question. Normalizing inside the branch, before the sub-layer, leaves the direct path untouched. Normalizing after the addition puts a normalization on that path at every layer, and deep models built that way usually need a learning-rate warmup to train.
In the Post-LN form, , the residual addition is still there; it is just normalized afterwards.
Example
The sub-layer's output is a small edit to the vector, not a replacement.
Where the course uses it
Related terms
- I — the identity matrixRead "I" or "the identity". It is a square matrix with 1s on the diagonal and 0s elsewhere. Multiplying by it leaves a vector unchanged.
- LayerNorm — layer normalizationRead "layer norm". It shifts and rescales each token's vector to have mean 0 and variance 1, then applies a learned scale and shift.