Transformer · Feed-forward Real generations, FFN toggled

The thinking
half-beat.

A Transformer block has two moves. Attention is communication — each token looks around and gathers what's relevant. Then the feed-forward network is computation — every token goes off and thinks on its own. Attention decides what to think about; the FFN does the thinking. Cut the FFN and watch what's left.

FFN on

Attention alone can't think. With the FFN off, the model can still move information between positions — but nothing processes it, so it collapses into noise. That two-step rhythm — gather, then think; gather, then think — repeated across every block is the whole engine. Next: stack these blocks into one brain.

uv run python 04_pretrain/ffn_extract.py # same model, same seed — only the FFN is switched

Real text from the trained model, written one character at a time — each character drawn at random from the model's own odds, from the same random seed both times · FFN bypassed by skipping its residual add · nothing faked. Built by AgentSure.