Every token compares itself with every earlier token, and each earlier token must be remembered.
Every token compares itself with every earlier token. For n tokens that is about n x n comparisons for each head in each layer. So doubling the input quadruples the work. A model also keeps each earlier token's keys and values in memory (the key-value cache, or KV cache), so it does not recompute them at every step.
Input length
Pairs compared
Key/value cache (7B-class model)
1,000
1x
about 0.5 GiB
4,000
16x
about 2 GiB
8,000
64x
about 4 GiB
32,000
1,024x
about 16 GiB
Doubling the input length quadruples the comparisons and doubles the memory.
The cache column uses a configuration of 32 layers, 32 heads of 128 numbers each and 2 bytes per number. The code computes it:
A test confirms exactly 2 GiB for 4,096 tokens. (The 7B-class configuration is typical, not a specific product.) If the model shares keys and values across groups of heads, so only 8 key-value heads are stored, the cache is four times smaller: for 16,000 tokens, 7.8 GiB with 32 heads and 1.95 GiB with 8. Modern models use this trick, called grouped-query attention (see the deep-dive).
Long inputs also have a quality cost. Models often use the beginning and the end of a long input better than the middle (module 7). So:
Price and delay rise with input size (module 6 covers this).
A large context window can hold everything, but cost and quality both suffer when it is stuffed. Retrieval (Level 4) exists to send only what is relevant.
How embeddings become search
You will use embeddings in Level 4 like this:
Turn each document chunk into a vector with an embedding model and store them.
Turn the user's question into a vector with the same model.
Return the chunks whose vectors have the highest cosine similarity to the question.
Never mix vectors from different embedding models in one index: their spaces are unrelated.
For the curious Why cost grows with the square
The score table has one row per token and one column per token, so with n tokens it has n x n entries. 1,000 tokens give a million scores per head per layer; 32,000 tokens give over a billion. Faster implementations avoid storing the whole table, but they still do the work. Some newer designs change the maths so cost grows more slowly with length, at some trade-off in quality.
What you will use daily at work
Practice
What it looks like
Embedding APIs
You call a provider's embedding model on text and get a vector back; you store it in a vector database.
Similarity search
A query vector is compared with stored vectors, and the closest chunks are returned.
Context budgeting
You keep prompts short because cost and delay grow with length, and you retrieve instead of pasting.
Prompt caching
Providers can reuse the work for an unchanged start of a prompt; keep that part identical across calls.
One embedding model per index
You record which embedding model built an index, and re-embed everything if you change it.
Draft: to be checked against current provider documentation (embedding models, prompt caching) before this page goes live.
Interview angle
Answer in your head first, then open it.
What is an embedding?
A learned vector where similar meanings are close; compare with cosine similarity.
Explain self-attention without maths.
Each token asks a question (query), compares it with every other token's label (key), and takes a weighted mix of their contents (value).
Why does a text generator use a causal mask?
So a token cannot see the future it is trying to predict.
Why is a long context expensive?
Pairwise comparisons grow with the square of the length, and the key-value cache grows linearly with it.
Why does a transformer need positional information?