Cosine learning-rate schedule

upshard

\eta_t = \eta_{\min} + \tfrac{1}{2}(\eta_{\max} - \eta_{\min})\left(1 + \cos\frac{t\pi}{T}\right)

Decays the learning rate along a cosine curve from its peak to its floor. It holds a high rate early, then eases off smoothly rather than dropping in steps, and it is the schedule nearly every large language model is trained under.

Symbols

$t$: The current step.
$T$: The total number of steps in the schedule.
$\eta_{\max}$: The peak learning rate, reached after warmup.

Save your progress

Your account

Delete your account?

Cosine learning-rate schedule

Keyboard shortcuts