MLP — multi-layer perceptron
A chain of layers, each a matrix multiplication followed by an activation function. The feed-forward network in every transformer block is an MLP with one hidden layer.
Hidden layers sit between input and output; their size is the network's width (section 4.2).
Where the course uses it
Related terms
- Neuron — a weighted sum and an activationThe smallest unit of a neural network: a weighted sum of its inputs plus a bias, passed through a fixed nonlinear function.
- FFN — the feed-forward networkRead "F F N of x". It is a small two-layer neural network applied to each token's vector on its own, with the same weights for every token.