Newton's method
Uses curvature as well as slope, jumping straight to the minimum of a local quadratic fit. It converges in dramatically fewer steps and is almost never used at scale, because inverting the Hessian is hopeless for billions of parameters.
- The Hessian of second derivatives.
- Its inverse, which rescales the step per direction.