GELU
Weights each input by the probability that a standard normal falls below it, giving a smooth curve that lets small negatives through instead of clamping them. It is the activation inside most transformer feed-forward blocks.
- The pre-activation input.
- The standard normal cumulative distribution function.