Training · The optimizer Eight real runs · the top rates really did break

Don't
blow up.

The gradient tells you which way to nudge every weight. It never tells you how far. That one number is the learning rate — and it decides whether training glides downhill, crawls forever, or detonates on the spot. Same model, same seed, same 600 steps. Only the step size changes. Drag it.

Learning rate0.002

Loss after 600 steps

—

Vs. the best rate

—

Survived

—

This is the most expensive dial in machine learning. A frontier run costs millions of dollars and takes months; if the learning rate is wrong, you find out days in, when the loss spikes and never comes back. So real training doesn't hold it fixed: it warms up from near zero, peaks, then decays along a cosine curve — big steps while the model is ignorant, small careful steps once it knows something. And a gradient clipping cap catches the one freak batch that would otherwise blow the whole run off the rails. The optimizer — the rule that turns a gradient into an actual step — is already doing its part here: every curve above was trained with Adam, which gives each weight its own step size and shrinks the ones that keep jittering. It helps a great deal, and it still does not save you from a rate that is simply too big.

uv run python 04_pretrain/optimizer_extract.py # trains the same model at 8 learning rates, records real loss

Every curve is a real training run: the same small model (2 blocks, 64 numbers per character) each time, started from identical random weights, 600 steps, loss read every 20 steps. At rates 0.5 and 1.0 the loss shot past 50 — far off any useful scale — and the recorder stopped there. Nothing is drawn after that step, because nothing after it was measured. Built by AgentSure.