04 · Pretraining / Scaling 9 real runs

Nine models.
One law.

The same GPT, trained nine times. Across → the model gets bigger — deeper, wider, longer context; downward ↓ it reads more text. Same code, same budget, same seed. Drag the slider and watch the grid resolve — brighter is better, and the corner it brightens toward is the whole story.

Training progress 100%
dim → bright = worse → better (lower loss) ✨ best cell ⚠️ overfitting — memorising the text, not learning it

Read it two ways. Down the grid — more text — quality climbs reliably. Across it — a bigger model — only pays off if you have the text to feed it. The top-right ⚠️ cell is the trap: the biggest model on the least text. Its samples actually read cleanly — but that's because it's reciting lines it memorised, not composing new ones. The tell is the number: on text it hasn't seen it bottoms out early at 2.36, then climbs all the way to 5.30 — worse than before it had trained at all. Looking good on what you've memorised is exactly the trap. That's the scaling law, and its catch, live.

How much text is enough

The rule none of these nine reaches.

Both dials cost money, so someone had to work out how to split a budget between them. Train hundreds of models at different sizes on different amounts of text, and the best pairings land on a line: roughly twenty characters of text for every parameter in the model. That result is called Chinchilla, and it is why nobody sensible builds a huge model and feeds it a little text. Divide each cell's text by its parameters and all nine runs above land on the same axis.

Two things this does not show. The rule counts the text a model reads once; every cell here reads its own text over and over until the steps run out, which is exactly how the starved ones end up reciting instead of composing. And these nine were each given the same number of steps, not the same amount of compute — so the ratio does not pick the winner here. What it picks is the disasters.