Alignment · Verifiable rewards Real GRPO run

Some answers
can be checked.

The last two stations both existed because nobody could say what a good answer was. One learned a taste. One got a list of nice words. Both were guesses, and one of them got gamed. But the reverse of honour is ruonoh, and no one has to have an opinion about that. Here the judge is arithmetic. Drag the rounds and watch what training against something that cannot be argued with actually buys you.

Two stations back
A model trained on which answer people preferred.
Guessed
One station back
A hand-written list of warm words to count.
Guessed
Here
One line of arithmetic. It is right or it is not.
Checked
The entire reward function of this station

That is all of it. No reward model. No human labels. No judge. Nothing here can be gamed, because there is nothing here to have an opinion.

The ceiling, before anything is trained

The tuned model from earlier answers reverse correctly — of the time when it always takes its top pick. But ask it 16 times and let it roll the die: how often is at least one of those 16 right? That second number is the roof. Reinforcement can only push a model toward things it already says sometimes.

Training round step 0
reverse, top pick
—
—
roof: any of 16 right
—
before training
one roll of the die
—
before training
reward on the training words
—
what the verifier paid out
groups with no signal
—
—

—

Held-out reverse accuracy, round by round
top pick, round by round roof (any of 16 right) where it started
—

—

Where the change lived

Reversing a three-letter word and reversing a six-letter word are not the same job. Split the held-out words by length and the gain — or its absence — stops being one number.

—

The tax

Only reverse was optimised. Three other tasks were never mentioned to the trainer — and were scored every round anyway, because a number that goes up while three others quietly go down is not an improvement. Each card reads their score before this station's training, then after it.

—

Why the group is the whole trick. For every prompt the model writes 16 answers, the verifier scores all 16, and each answer is judged against its own group's average — better than its siblings, push it up; worse, push it down. That group is what lets this run skip the second network that ordinary policy gradient training needs to guess how good a position is, and it is where the field's name for this comes from: GRPO. When all 16 siblings score the same, the group says nothing at all and contributes nothing. The counter above shows how often that happened. Training against a reward a verifier settles rather than a judge estimates is what the field calls RLVR.

What this station cannot show you. The answer format is Q: reverse word / A: answer — there is nowhere to write working out. So no amount of this training can grow a model that thinks before it answers; it has no room to think in. And the whole vocabulary this model was ever trained on contains —, so a sum cannot even be written down here, let alone learned. The famous version of this method is run on maths. This one is run on the one checkable thing this corpus can express.

—