Serving · Streaming attention Real keys and values, exact memory

The table nobody
should write down.

To let every position look at every other position, the textbook builds a score for each pair — a square table. Double the context and it quadruples. At the lengths a real assistant runs at, that one table is larger than the entire model, and it is built and thrown away for every single layer. The bottleneck was never the arithmetic. It was carrying that table around.

First, how bad does it get?

Exact arithmetic for one layer of the small model on this page — its four heads, four bytes a number. That model only ever sees 128 characters; drag past that and you are watching the same arithmetic carried out to the lengths real assistants run at.

context length
the whole table
every pair of positions, written down at once
one tile at a time
eight positions' worth of that table — all the streaming version ever holds, then throws away

Now, how do you get the same answer without it?

Softmax looks like it needs the whole row — you have to divide by a total you can't know until you've seen everything. You don't. Keep a running maximum and a running total, and rescale what you already have whenever a bigger number turns up. Walk it.

running maximum
—
running total
—
rescaled what it had by
—
positions read
0

What this page does not measure: speed. Streaming like this in Python is slower than building the table, because the win isn't in the arithmetic — it's that a fused kernel can keep each tile in the small, fast memory right next to the arithmetic units, instead of shipping a giant table out to the card's main memory and back. That kernel isn't in this repo, so there is no stopwatch here. The memory is exact, the answer is checked, and the speed is left to people who can actually measure it.

This is the same idea as the last station, one level down. The cache stopped the model re-reading what it had already written. This stops it from ever writing the working-out down in the first place. Neither one changes a single answer — and that is the point. Everything in this act is about making the same model cheap enough to actually serve, without touching what it says.