Inference · KV cache Real timings, both runs identical

The typewriter
is real.

A model writes one character at a time — and to write the next one it has to read everything written so far. Do that honestly and each character costs more than the last. The fix is the plainest idea in this whole river: don't read it twice, keep the notes. Those kept notes are called a KV cache; the switch below throws them away.

notes kept

characters written
0
time spent
0.0 ms
this character cost
—
attention scores computed
0
The cache · one column per character written
Milliseconds to write each character
no cache — climbing kv cache — flat

The honest gap. Keeping the notes cut the attention arithmetic by a factor of — but the wall clock only improved by . Both numbers are real, and the difference between them is the lesson: at 120 characters the context is still short, so the re-reading is not yet the main cost (double the length and that re-reading is four times the work), and plenty of the remaining time is work the cache can't remove — the per-character thinking still has to happen, every time. Don't fall for the big ratio. Look at the shape of the lines instead.

Nothing is free. The cache has to live somewhere: numbers per character, or for a full context window on this tiny model. Multiply that by a real model's size and a real conversation's length and you have the thing that actually decides how many people a server can talk to at once. Next: the same pressure, one level down — where the memory traffic really goes.

That number was a choice

Several heads, one set of notes.

Numbers kept per character
A full window, characters
Size of the model itself
Held-out loss, steps

The other half of the same problem

Stop asking it one character at a time.

Characters guessed ahead 1
Big-model passes
Characters per big-model pass plain decoding gets 1.00
Guesses that survived the check
Both models, in big-model passes

And the catch, counted honestly.