Where training data comes from, and what can go wrong.
Data
How much? In the example, question side only:
Training pairs
New questions
General language
10
4%
32%
20
7%
35%
40
20%
40%
72
48%
50%
With too few pairs the model memorises the pairs it saw, forgets general language, and gains nothing on new questions. Your numbers will differ; the lesson is that the gain needs coverage: every term the model should learn must appear in several training pairs.
Where pairs come from:
Logs and feedback: questions with the page a user clicked, or the document a support agent attached. The most realistic data, and biased toward what your current system already shows.
Existing pairs: ticket and resolution, title and body, question and accepted answer.
Generated questions: a language model writes plausible questions for each passage. Cheap and wide, but they sound like a model, not like your users; check a sample by hand.
Labelling with a stronger model (cross-encoder or language model) to expand a small labelled set.
Split by document or topic before training, not by random row, or near-copies of training pairs leak into the test (module 7). Keep a held-out set and a general-language canary set that you never train on.
Risks and defences
Risk
What it looks like
Defence
Forgetting
general and out-of-domain questions get worse
gentler learning rate, mix in general pairs, keep the tuned part small (adapter), a canary set
Memorising
near-perfect training score, much lower on new questions
held-out validation by topic, more varied data, fewer epochs
Narrow transfer
no gain on topics absent from training
cover the vocabulary; plan retraining when the domain grows
False negatives
mined "hard negatives" teach wrong distinctions
score thresholds or rank windows, human check on a sample
Evaluation leakage
great test score, disappointing real use
split by topic, test on logged real questions
Maintenance
the domain changes, the model does not
version the model, schedule re-evaluation, re-embed through blue-green (module 9)
What worked best in the example (question side only; new questions and general language):
Setting
New, same problems
General language
default (learning rate 0.02)
50%
51%
gentler (learning rate 0.005)
55%
72%
pull the matrix back toward the original (weight 0.01)
40%
66%
replay 20 general-language pairs
47%
77%
The cheap fix was a gentler learning rate, which was better on both columns than the default, and mixing in general pairs protected general ability most. Pulling toward the original cost domain accuracy. Differences of a few points between rows are within run-to-run noise.
New-question accuracy (dark bars) against general-language accuracy (light bars) for five settings
Practical checklist
Exhaust cheaper fixes and measure the gap on real questions first (module 7).
Build the data: pairs with coverage of your vocabulary, a topic-level split, a general canary set.
Start from a pretrained embedding model and a small learning rate; consider a question-side adapter first.
Compare against the untuned model on the same held-out questions, including general ones.
Mine hard negatives carefully and check a sample for false negatives.
Plan the rollout: second index, golden questions, switch both sides together, keep the old one (module 9).
Re-evaluate on a schedule; version the model with its data.
Interview angle
"Our retrieval misses questions because users and documents use different words. Do we need a custom embedding model?" Probably not yet: first check chunking, add keyword search, try query rewriting or a reranker, and measure what remains on real questions. If a vocabulary gap persists and you have thousands of (question, passage) pairs, fine-tune a pretrained embedding model with a contrastive loss and in-batch plus carefully mined hard negatives, and evaluate against the untuned model on held-out questions split by topic, plus a general-language canary set because tuning can degrade general ability. Mention operations: if you tune both sides every vector must be re-embedded into a second index and switched together; a question-side adapter avoids that. Close with the limits: it will not transfer to topics absent from the training data, so you must maintain it as the domain changes.
Check yourself
Training accuracy reaches 100 percent while accuracy on general questions falls. What do you check before shipping? (The held-out domain questions the model never saw, and the general set. A rise on training data proves nothing. A fall on general questions is the cost to weigh.)
For the curious How ranks are counted
The evaluation ranks a document by counting how many documents score strictly higher than the right one, so a tie counts in the model's favour. With real, continuous scores ties are rare, but if two documents have identical text they share a score, so remove duplicates from a test corpus before you evaluate.
What you will use daily at work
Practice
What it looks like
Trying cheaper fixes first
Better chunks, hybrid search, query rewriting and a reranker come before any training.
A held-out test set
Questions the model never trained on, plus a general set to catch forgetting.
Library support
Sentence-embedding libraries provide the loss and training loop; the work is the data.
Draft: confirm current library names and training options before starting.