Training · Backpropagation One real forward, one real backward

Where the
blame goes.

The model just read some Shakespeare and guessed the next character, over and over, and got a lot of them wrong. One number says how wrong. The hard part isn't measuring that — it's working out which of its 1.8 million weights — the adjustable numbers that are the model — to blame, and by how much. Watch the blame fall backward through the stack. Sending it back like this is called backpropagation, and the push one weight is told to take is its gradient.

First — what is a weight? One adjustable number, sitting in one slot of one matrix. That is the whole of it, and this model is — of them and nothing else — — of them inside the repeated blocks you have already met, the rest being the character table, the position tags and the output scorer. Training does exactly one thing to a weight: nudge it up, or nudge it down. Nobody chose their values. Somebody did choose how many there would be — by picking how many blocks to stack and how wide to make them — and no amount of training changes that. Those are settings; these are weights. Below, every one of them is about to be handed an instruction.

The batch it just got wrong
how wrong, in one number:
weights told which way to move0
Actual gradient numbers
Nothing has been blamed yet. Each stage below holds one number per weight — this is where you see the raw ones.

So was the blame any good?

Every weight now has an instruction. Nudge each one a little in the direction it was told — weight − 0.0003 × its own gradient — and measure the very same batch again.

wrong before
wrong after one step
improvement

That is the whole of learning. It got less wrong by about five thousandths. This model was given four thousand steps exactly like this one, each on a fresh batch, and that is what turned its output from noise into Shakespeare.

Notice the blame doesn't fade. Read the pushes down the stack: the character meanings at the very bottom — furthest from the mistake, eight sub-layers away — got the largest average push of anything, bigger than the output head sitting right next to the loss. Nothing is dying on the way down. You may have heard that deep networks suffer from vanishing gradients. They can — but that needs far more depth than the four blocks here, and we're not going to stage it for you. This model simply doesn't have the problem, and the numbers above are what it actually reported. Next: hand these same gradients to a real optimizer.