Part 1 of 7 · 1 min read
Overview
In this module you will
- Explain an embedding as a position in space where meaning is a direction, and compare two embeddings with cosine similarity.
- Describe self-attention in plain words, and compute it in about ten lines of NumPy.
- Explain why a text generator hides the future from each token, and why a model needs position information.
- Say why long inputs are expensive, and estimate the memory a long conversation needs.
Embeddings are the engine behind search and retrieval (Level 4): you will turn documents and questions into lists of numbers and compare them. Attention is the reason the context window exists, the reason it has a price, and the reason a model can use an instruction or a fact placed anywhere in it.
Knowing the mechanism, not the maths, lets you predict when a long prompt will cost more, when a model may "lose" something in the middle, and why sending only the relevant text beats pasting everything in.