Transformer · Position Real vectors, measured inside the trained model

Order-
blind.

Attention builds each token's new vector by averaging the tokens it can see — and an average doesn't care what order things arrived in. So "the king is dead" and "is the king dead" are the same bag of characters. The fix is almost silly: staple a position tag onto every token before the model ever looks at it. Positional encoding, the field calls it. It arrives with that tag switched off. Turn it on below and watch what the model becomes.

Sentence A — the real order

Sentence B — same characters, shuffled

Position tags

The model reads the sentence in four passes, one after another; each pass is a block. The number after each pass is how far apart the two sentences look inside the model — 0 means it cannot tell them apart at all, bigger means further apart.

Tags on — the model you trained

Tags off — same weights, no positions

How the tag is actually made. This model uses the plainest version there is: a table with one tag per slot — slot 1 has its own, slot 2 another — and training decides what is in them. The first big models built the tag by hand instead: a fingerprint of sines and cosines at different frequencies, unique for every slot. Today's models mostly use RoPE, which doesn't add a tag at all — it rotates each token's vector by an angle proportional to its position, so that when two tokens are compared, only how far apart they sit shows up. Which raises the question the table can't answer, and the rest of this page measures.

The bill for the tag

Where does a million-token context come from?

How far to read

lookup table rotation (RoPE) loss at that slot · lower is surer

Table model — the position table

Rotation model — the position table

uv run python 04_pretrain/position_extract.py   # reads the last token's vector after every block, tags on vs off
uv run python 04_pretrain/rope_extract.py       # trains a table model and a rotation model side by side, then reads both past the edge

Every number is measured inside the trained character-level model: the largest disagreement across all 192 numbers of the final token's vector, after each block. The four orders are exact character permutations of one another and end on the same character, so order is the only thing that can differ. Built by AgentSure.