Part 1 of 5 · 3 min read
Overview
In this module you will
- Decide whether fine-tuning an embedding model is justified, and what to try first.
- Explain contrastive training: pairs, in-batch negatives, hard negatives, and the loss that uses them.
- Choose where to apply the tuning (question side only, or both) and say what each choice does to your index.
- Predict and measure the risks: memorising, forgetting general language, no transfer to unseen topics.
- Build the evaluation that tells you whether the custom model is actually better.
Why this matters for agents
Everything earlier in this level works around a general-purpose embedding model: hybrid search patches its blind spots, query transformations reword the question, rerankers fix the order. Sometimes none of that is enough, because your domain's words mean something the general model does not know. A ticket says "the app freezes", the manual says "deadlock"; a lawyer's "consideration" is not the everyday word. A model trained on your question and answer pairs can learn those links. It is also the most expensive and riskiest change in this level, so this module is about doing it last, and doing it measured.
About the examples
The "embedding model" here is a toy: every word has a fixed pretrained vector (ordinary synonyms like buy and purchase start close; domain jargon does not), a text is the normalised sum of its word vectors, and the only trainable part is a 64 by 64 matrix per side (questions and documents) learned with the same loss real systems use. The domain is an invented support desk with 36 problems, where customers and manuals use different words for 24 of them, and 160 one-line manual pages. Real models have hundreds of millions of parameters and are tuned very differently, so do not carry any percentage over; what carries over is the shape of the results: the gain, the gap between training and new questions, the loss of general ability, and the cost. All numbers are means of five runs with different random splits. No real model was trained.
Try this first
The ladder, cheapest first:
- Fix chunking and metadata (module 1).
- Hybrid search (module 3): keywords catch codes and names that embeddings blur.
- Query transformations (module 4): rewriting or expanding the question can bridge a vocabulary gap without training.
- A reranker (module 3): a stronger model reorders the shortlist.
- Measure (module 7): do you have a gap that survives all of this, on questions your users actually ask?
- Only then, fine-tune the embedding model, or put an adapter on it.
The gap, in numbers
One training example from the toy domain:
customer asks : again reset when stuck
manual says : procedure affected provisioning systems livelock
The two share no words, because "reset" is "provisioning" and "stuck" is "livelock" in the manual. Of the 36 problems, 24 are like that; 12 use the same word on both sides. Searching all 160 pages with a customer question, the untuned model ranks the right page first for 19% of the new questions about those problems. A fine-tuned model reaches 50 to 70% (section 5). That is the whole promise: it learns that "stuck" belongs near "livelock".