02 · Attention Real weights from a trained model

Words read
each other.

A word's meaning depends on the words around it. Attention is the mechanism: to process each word, the model looks back over the sentence and splits 100% of its attention across the words so far — the word it is standing on included — deciding which of them matter right now. These are the real weights, over the very same sub-word tokens from the tokenizer — pick a line, a layer and a head, then grab a token and watch who it reads.

Sentence
Layer
Head
Spotlightdrag a word — watch it read

Full grid — every word reading every word

Two things worth noticing. First, it can only look backward — grab a word in the middle and everything to its right goes dark, because when you're predicting the next word you haven't seen the future yet. Second, switch heads: they specialise. One tracks the word just before, another hunts further back. Stack these heads block on block and you have most of a Transformer — what is missing is the step where each word thinks about what it just gathered. That step is the next stop.