GELU, ReLU, Swish — activation functions

$\operatorname{ReLU}(z) = \max(0, z)$ has a sharp corner at zero and is completely flat below it, so negative inputs get no gradient. $\operatorname{GELU}(z) = z \cdot \Phi(z)$, where $\Phi(z)$ is the probability that a standard normal random number is less than $z$. The result is a smooth curve that lets small negative values through. Gradients behave better, at a little more compute.

Swish, $z \cdot \operatorname{sigmoid}(z)$, has a very similar shape. SwiGLU uses two branches: one passes through Swish, the other stays linear, and the two are multiplied entry by entry, so one branch gates (scales up or down) the other.