Dropout

Deep learning · Formula · Appears in 1 lesson

h~=m⊙h1−p,mi∼Bernoulli(1−p)\tilde{h} = \frac{m \odot h}{1 - p}, \qquad m_i \sim \mathrm{Bernoulli}(1-p)

Randomly zeroes activations during training so no single unit can be relied on, then scales the survivors so the expected output is unchanged. At inference nothing is dropped, and the scaling is what makes the two regimes agree.

Symbols

mm
A random binary mask, redrawn every step.
pp
The probability a unit is dropped.
⊙\odot
Element-wise multiplication.
Esc
↑↓ navigate↵ open? shortcuts

Keyboard shortcuts

Press Esc to close

Anywhere

⌘K
Search and commands
/
Search
?
This sheet
Esc
Close

Reading a lesson

J
Next lesson
K
Previous lesson
B
Bookmark