Dropout
Randomly zeroes activations during training so no single unit can be relied on, then scales the survivors so the expected output is unchanged. At inference nothing is dropped, and the scaling is what makes the two regimes agree.
- A random binary mask, redrawn every step.
- The probability a unit is dropped.
- Element-wise multiplication.