RMSProp
Gives every parameter its own learning rate by dividing by the recent size of its gradient. Parameters with consistently large gradients get smaller steps, which keeps badly scaled problems from stalling.
- A running average of squared gradients.
- Its decay rate.
- A small constant that prevents division by zero.