04 · Pretraining A real run, replayed step by step

Train your own
mini LLM.

Everything so far was one model's parts. Now watch them learn. Press the button and a real GPT trains on Shakespeare, character by character — back to the simplest tokens, so you can watch it learn from scratch. The loss you see is the loss it recorded; the text on the right is what it actually wrote at that step. It starts as noise. Give it a few thousand steps.

Loss—

What it writesstep 0

Replay

Press Train. It really starts from noise.

You just trained a language model. A real one: a GPT with parameters, every one of them tuned by those four thousand steps. Nothing faked. Nothing told it English; it only ever guessed the next character and got corrected. Out of that one dumb loop came grammar, names, and the layout of a play — a name in capitals, a colon, then the lines. Make it bigger and feed it more — that's the next step, and the whole trick behind ChatGPT.

uv run python 04_pretrain/train.py --data data/input.txt --label B_large_large --n_layer 4 --n_head 4 --n_embd 192 --block_size 128 --max_iters 4000   # the exact run replayed above

Character-level GPT · 1.83M params · 4000 steps on Shakespeare · every loss and sample real, nothing faked. Built by AgentSure.