ReLU
Pass positives through, clamp negatives to zero. Its gradient is one or nothing, so it neither saturates nor vanishes, and that single property is most of why very deep networks became trainable.
- The pre-activation input.
- Selects the larger of the two arguments.