Serving · Quantization Real held-out loss, real text

How few bits
can it survive on?

This model is nothing but a pile of numbers — its weights. Each one is written down using 32 bits: 32 on-off switches, enough to hold a value to about nine decimal places. Almost all of that care is wasted, and you can see it in the weights themselves: —. That is the whole range this model ever needed, held to nine decimal places. Round every weight onto a coarser set of allowed values and the file gets dramatically smaller — that rounding is called quantization. Turn the dial down and find the point where it stops being a language model.

bits per weight

one scale per row

model on disk

held-out loss

verdict

What it writes now — greedy, so this is the weights and nothing else

Held-out loss against bits per weight — the line going up means worse

one scale per row one scale for the whole matrix guessing at random

Eight bits is very nearly free. Quarter the storage, and the loss moves by — you would never notice. That is why almost everything you run locally is served this way. Four bits costs you something real but still writes English. Below three, there is a cliff, and on the other side of it the file is smaller than a photograph and can't write a sentence.

Two bits is past the end. Both scaling flavours score worse there than a model that has learned nothing at all and simply guesses evenly among the 65 characters (). So don't read the 2-bit row as one method beating the other — comparing two corpses isn't a result. It's only above the cliff that the comparison means anything, and there one scale per row buys better answers.

It isn't free, though. Every step size has to be stored too, or the file can't be read back: of them instead of , which is a flat either way you turn the dial. Flat is the interesting part. The weights shrink as you squeeze them and that cost doesn't, so the same is of the file at 8 bits and of it at 3 — the harder you squeeze the model, the more of what's left is bookkeeping. Flip the switch and watch the size meter move.

What this page does not measure: speed. Quantization is fast in the real world because integer arithmetic is fast, and that needs kernels this little repo doesn't have — here the weights are rounded and put back into floats to do the maths, which is the standard way to study quality. Storage is exact arithmetic and the losses are measured. A stopwatch would have been a lie, so there isn't one. Next: it stops continuing your text and starts answering you.