ReLU

Deep learning · Formula · Appears in 2 lessons

ReLU(x)=max⁡(0,x)\mathrm{ReLU}(x) = \max(0, x)

Pass positives through, clamp negatives to zero. Its gradient is one or nothing, so it neither saturates nor vanishes, and that single property is most of why very deep networks became trainable.

Symbols

xx
The pre-activation input.
max⁡\max
Selects the larger of the two arguments.
Esc
↑↓ navigate↵ open? shortcuts

Keyboard shortcuts

Press Esc to close

Anywhere

⌘K
Search and commands
/
Search
?
This sheet
Esc
Close

Reading a lesson

J
Next lesson
K
Previous lesson
B
Bookmark