00 · Tokenizer Real BPE, runs in your browser

Text becomes numbers.

You just watched it guess letter by letter — the simplest version. Big models don't read letters; they first chop text into tokens and hand each one an ID. This tokenizer was trained on Shakespeare: it learned that chunks like ing, ·the and ·lord show up so often they deserve to be one token each — the · stands for the space in front of a word. Type below and watch it split. The mini model further downstream stays on single letters — 65 of them — because that is small enough to watch learn from nothing. The idea is identical; only the size of the pieces changes.

Your text
Tokens each chunk is one token, and the number under it is that token's place in the tokenizer's list, not a score. · is a space, ⏎ a line break.
0Characters
0Tokens (BPE)
0Chars / token

The same sentence, cut two ways — every gap is one more piece to pay for
a piece per character 0
this tokenizer 0

How one word gets built

Starting from raw characters, BPE keeps merging the most frequent pair until only known tokens remain — here's question:

Why bother?

Character-level is simplest but wasteful — the model spends effort re-learning that t-h-e is a word, everywhere. Word-level can't handle words it never saw. BPE sits in between: common words become single tokens, rare ones fall back to smaller pieces down to single characters. Every real LLM tokenizer works this way — just with a far bigger vocabulary, and a byte-level fallback so nothing is ever truly unknown.

The whole trick: once text is a list of token IDs, the model never sees language again — only numbers. Everything downstream — meaning, attention, prediction — happens on these IDs. Get this wrong and nothing else can work.