Activation function — the nonlinearity
The fixed nonlinear function applied after a layer's weighted sums, such as ReLU, tanh, GELU or Swish.
Without it, stacked linear layers collapse into a single linear layer, so depth would add nothing (section 4.2).
Where the course uses it
Related terms
- Neuron — a weighted sum and an activationThe smallest unit of a neural network: a weighted sum of its inputs plus a bias, passed through a fixed nonlinear function.
- GELU, ReLU, Swish — activation functionsThe nonlinearity applied inside the feed-forward block. ReLU sets negative numbers to zero; GELU and Swish do almost the same but with a smooth curve.