d_{\text{ff}}

d_ff — the feed-forward width

Read "d f f". It is the width of the hidden layer inside the feed-forward block: traditionally 4 × d_model, and about 8/3 × d_model in gated versions like SwiGLU.

The feed-forward block widens each token's vector to $d_{\text{ff}}$ numbers, applies a nonlinearity, and narrows it back to $d_{\text{model}}$ . These two large matrices hold most of a transformer's parameters.

SwiGLU uses three matrices instead of two. To keep the parameter count the same, $d_{\text{ff}}$ is cut to about $\frac{8}{3}d_{\text{model}}$ : three matrices of width $\frac{8}{3}d$ hold as many numbers as two of width $4d$ . This keeps comparisons between the two designs fair.

Example

d_{\text{model}} = 768 \;\Longrightarrow\; d_{\text{ff}} = 4 \times 768 = 3072

GPT-2 small.

Where the course uses it

15.4 SwiGLU · Module 15, The LLaMA Blockfirst use

Related terms

d_model — the model's widthRead "d model". It is how many numbers represent one token inside the network: 768 for GPT-2 small, 4096 for LLaMA-3 8B.
FFN — the feed-forward networkRead "F F N of x". It is a small two-layer neural network applied to each token's vector on its own, with the same weights for every token.