03 · Transformer A real run, replayed step by step

The whole engine.

Stack the pieces and you have the Transformer: embed the tokens, run them through a few identical blocks — each one attention plus a small network, wrapped in residual shortcuts — then read off a probability for the next token. Press Predict and step through the run it recorded: the whole stack, one character at a time.

The characters so far
each one is just its number in the alphabet
↓
Give each one meaning, and a position
every character becomes 192 numbers — a width someone chose
↓
× 4 blocks
Look around, then think
each character reads the earlier ones
add what it learned back on
each one thinks on its own
add what it concluded back on
↓
Score every character it could write next
one score for each of the 65 letters it knows
↓
Turn those scores into percentages
the bars on the right →
Generated so far
Next character · the model's real prediction
Everything on the left produced this one distribution. Pick from it, append, and run the entire stack again — that loop is how all text is generated.

That's the whole idea. The residual shortcuts let information (and gradients) skip straight past each block, which is what makes stacking dozens of them trainable. And notice the loop: the model only ever predicts one next token, then eats its own output and runs again. Everything you've read from an AI was written exactly this way — this model just has 1.8M parameters instead of a trillion.

The parts of the block nobody mentions

What the steadying step is for.

The obvious way to find out is to switch them off in the model you just ran. That is the wrong way, and this river has already made the mistake once: station 8 pulled the residual shortcut out of a trained model and got an answer that looked decisive and was worthless, because the weights had spent their whole training growing around the thing being removed. So instead, two models were trained from scratch in one process — same seed, same text, same held-out set, same number of steps. The only difference is those modules.

with them without them held-out loss · lower is surer
with them, after 3,000 steps
without them, after the same 3,000 steps