Learning Rate Warmup & Decay ·
α(t) Schedule
un
seel
.com · linear warmup · cosine decay · step size
step
0
/
100
α(t)
0.000
loss
1.00
phase
warmup
warmup: α 0 → α_max
peak α_max
cosine decay → α_min
ball descends loss surface
▶ Play
←
→
🔇 Unmute
Reset
Un
seel
.com · Learning Rate Warmup & Decay