Keyword and vector search: Hybrid search and reranking

Keyword search: BM25

BM25

A scoring formula for term-based search. A passage scores higher when it contains the query's words, when those words are rare across the collection (a rare word such as e-4021 is strong evidence, a common word such as plan is weak), and when a word repeats but with diminishing returns. It also adjusts for passage length so long passages do not win just by being long.

For each query word, BM25 adds idf × f(term frequency, length), where idf (inverse document frequency) is large for rare words. The example implements it in about twenty lines (search.py). A test checks that e-4021 has a higher idf than plan.

BM25(DOCS).rank("E-5503")[0]                                   # 'd02', the passage about that error
BM25(DOCS).rank("my money back for the yearly subscription")   # does not contain 'd06' (the refund passage)

Good at: exact identifiers, names, error codes, function names, quoted phrases, anything where the exact word is the signal. It is fast, needs no model, and is easy to debug (you can see which words matched). Weak at: paraphrase and synonyms ("money back" versus "refund"), and it returns nothing for words it has never seen.

Vector search, in one paragraph

Module 2 covered it. It finds passages whose meaning is near the query's, so it handles paraphrase, but it can miss exact tokens and it always returns something, even when nothing is relevant. In the example, Dense(DOCS).rank("my money back for the yearly subscription")[0] is the refund passage, which BM25 missed.

Hybrid search: ask both, then merge

Hybrid search

Running keyword search and vector search for the same query and merging their results into one ranking.

The two retrievers score on different scales (BM25 scores are unbounded; cosine similarity is between −1 and 1), so you cannot simply add the scores. Two common ways to merge:

Reciprocal rank fusion (RRF). Ignore the scores and use only the ranks. Each list gives a passage 1 / (k + rank) points; add the points across lists. k (often 60) softens the advantage of first place.

rrf([bm25_ranking, dense_ranking], k=60)

Example by hand with k = 60: passage A is 1st in the keyword list and 3rd in the vector list, so 1/61 + 1/63 = 0.0164 + 0.0159 = 0.0323. Passage B is 2nd in both, so 1/62 + 1/62 = 0.0323. Almost a tie, and a passage that appears in only one list gets only one term. RRF therefore rewards agreement between the retrievers and needs no tuning of scales.

Weighted score fusion. Rescale each score list to 0 to 1 (min-max), then compute alpha × dense + (1 − alpha) × sparse. You get a knob (alpha), but you must trust the rescaling, which is sensitive to outliers.

On this toy corpus both give the same results (94 percent recall at 3, 78 percent top-1). Do not conclude they always agree; exercise Medium 2 explores it.

Reranking: a second, more careful look

Reranker

A model or function that takes the query and a shortlist of candidate passages, scores each (query, passage) pair together, and returns them in a better order. A bi-encoder (the usual embedding model) embeds the query and passage separately and compares vectors. A cross-encoder reads the query and passage together, so it can notice that two words are close, negated or contradicted. It is more accurate and much slower, so it is used only on the shortlist.

The standard two-stage shape. The first stage is cheap and wide; the second is expensive and narrow.

The toy reranker (ToyReranker) rewards covering every query term (as a word or a concept) and having the terms close together in the passage. It is not a learned model, but it shows the key properties:

It sees query and passage together, so it can use proximity, which neither retriever uses.
It costs one evaluation per candidate, so the cost grows with the shortlist and not with the whole collection (the demo scores 72 pairs for the 18 questions, against 324 if it scored every passage).
It can only reorder what it is given. A passage the first stage did not return stays missing.

Real output of evaluate.py. The best retriever depends on the kind of question; the hybrid is safe on all of them; the reranker lifts first place.

Reading the chart (toy data):

Exact terms: keyword search beats vector search (83 percent versus 50 percent first place), because the toy vector model blurs digit-bearing codes.
Paraphrases: vector search beats keyword search (67 percent versus 50 percent), because keyword search shares no words with "money back".
Hybrid matches the better single method on exact terms, beats both on paraphrases (83 percent), and beats either overall (78 percent). It is not free of cost: on mixed questions first place slipped to 67 percent against keyword search's 83 percent, because the vector side pulled a wrong passage up ("Starter plan limit" ranks the right one second). On "lag is high for my requests" the right passage lands at rank 4, since each list disagrees about it.
Reranking the fused top 5 reaches 94 percent first place. The one remaining miss is the question where the passage was ranked fourth and the reranker's simple rules could not tell it apart from the others.