\operatorname{FFN}(\mathbf{x})

FFN — the feed-forward network

Read "F F N of x". It is a small two-layer neural network applied to each token's vector on its own, with the same weights for every token.

In the original Transformer paper, $\operatorname{FFN}(\mathbf{x}) = \max(0, \mathbf{x}W_1 + \mathbf{b}_1)W_2 + \mathbf{b}_2$ : multiply, add a bias, set negatives to zero, then multiply and add a bias again. Modern models replace the $\max(0, \cdot)$ with GELU or SwiGLU.

It is called position-wise because tokens do not exchange information here. Attention moves information between tokens; the FFN transforms each token by itself. The nonlinearity in the middle is essential: without it, $W_1$ followed by $W_2$ would be the same as a single matrix $W_1 W_2$ , and the extra layer would add nothing.

Where the course uses it

7.4 The Feed-Forward Network · Module 7, The Block and the Full Modelfirst use
11.4 Tensor Parallelism · Module 11, Training Across Many GPUs
17.2 Many Small Experts · Module 17, Mixture of Experts

Related terms

d_ff — the feed-forward widthRead "d f f". It is the width of the hidden layer inside the feed-forward block: traditionally 4 × d_model, and about 8/3 × d_model in gated versions like SwiGLU.
GELU, ReLU, Swish — activation functionsThe nonlinearity applied inside the feed-forward block. ReLU sets negative numbers to zero; GELU and Swish do almost the same but with a smooth curve.
Scaled dot-product attentionEach token scores every other token, turns the scores into weights that add up to 1, and takes the weighted average of their value vectors.