Scale · Tensor parallelism Real split, checked against the undivided answer

One model,
cut across machines.

Past a certain size a model does not fit on one card, and no amount of patience fixes that. So you cut it up. Not the work — the model itself: each machine is handed a slice of every layer and computes a partial answer, and the partial answers are added together. Every machine holds less. None of them can finish alone.

machines

across the wire

sums per character

This is one laptop, and it says so. There are no eight cards here and nothing on this page pretends otherwise. What has actually been done is the arithmetic: the real weight matrices really were cut up, each shard's partial answer really was computed separately and added back, and the result really was compared against the whole model. What a rack of real machines adds is the wire between them — and the fact that no shard can take a step until every other shard has finished and reported in. That coordination is the entire difficulty of the thing, and it is the one part a single machine cannot show you.

The trade never goes away. Double the machines and each one holds a little over half as much — but the traffic between them doesn't shrink, and every sum is a moment where the fast machines sit idle waiting for the slow one. That is why these racks are wired together with interconnects that cost more than the computers, and why "just add more GPUs" stops working long before you run out of money.