Stochastic gradient descent
The same step, but with the gradient estimated from a small batch instead of the full dataset. The noise this introduces is not purely a cost — it helps escape sharp minima, and it is the only reason training at scale is affordable.
- A minibatch sampled from the training set.
- The loss averaged over that batch only.