Adam
Momentum and RMSProp in one optimiser: a running mean of gradients divided by a running mean of their magnitudes, both bias-corrected for their cold start. It is the default for transformer training, and the two extra states are why it costs three times the memory of plain SGD.
- Bias-corrected first moment — the mean gradient.
- Bias-corrected second moment — the mean squared gradient.
- A small constant for numerical stability.