The transformer block

Modules 1 and 2 showed how text becomes vectors and how attention lets tokens look at each other. A real model repeats one unit over and over. This section builds that unit.

The parts, in order:

• Layer normalisation rescales each token's vector to zero mean and unit spread, which keeps the numbers stable through many layers. A test checks mean 0 and variance 1 (real output on random numbers with mean 3 and spread 5: means of 0.0 and variances of 1.0).
• Causal multi-head attention is module 2, run in several heads in parallel and mixed by an output matrix. The mask means a token never sees later tokens.
• MLP (multilayer perceptron, also called the feed-forward network): a small two-layer network applied to each token separately. It widens the vector to four times its size, applies a smooth curve called GELU, then shrinks back. GELU is close to zero for negative inputs and close to the input for positive ones (real values: GELU(-3) = -0.004, GELU(-1) = -0.159, GELU(0) = 0, GELU(1) = 0.841, GELU(3) = 2.996). Much of a model's stored knowledge is thought to live in these layers, but that is an active research question, not a settled fact.
• Residual connections: the output of each part is added to its input, instead of replacing it. They let information, and the learning signal during training, skip past layers. A test proves it: with every weight set to zero, the block returns its input unchanged.

def transformer_block(x, p, rope_positions=None):
    x = x + causal_multi_head(layer_norm(x), p, rope_positions)
    return x + gelu(layer_norm(x) @ p["w1"]) @ p["w2"]

Read it: normalise x, run attention on it, and add the result back to x. Then normalise again, run the MLP, and add that back. "Pre-norm" means the normalisation comes before each part.

Two tests are the important ones. Shape: the output has the same shape as the input, which is what makes stacking possible. Causality: change token 4, and tokens 0 to 3 produce identical outputs while tokens 4 and later change.