All chapters
  1. Start hereA model knows only what it can see now
  2. Foundations · 1The context window: tokens, messages and order
  3. Foundations · 2Signals and authority: who is allowed to instruct the model
  4. Foundations · 3The context assembly pipeline: select, rank, deduplicate, pack
  5. Retrieval · 4Retrieval fundamentals: lexical search, embeddings, hybrid search, reranking
  6. Retrieval · 5RAG architecture: chunks, metadata, grounding, citations, freshness
  7. Budget · 6Context budgeting: truncation, dilution, token cost and caching
  8. Budget · 7Compression, summarization and compaction, with loss tests
  9. Agents · 8Coding-agent context: repository maps, instruction files, diffs and tool outputs
  10. Agents · 9Working context, durable memory and checkpoint state
  11. Agents · 10Dynamic context: just-in-time retrieval, selective tool disclosure, delegation
  12. Agents · 11Multi-agent context isolation and scoped handoffs
  13. Trust · 12Context security: injection, poisoning, access controls, tenant boundaries
  14. Trust · 13Measuring context quality: recall, utilization, grounding, ablation
  15. Practice · 14Frameworks and standards: what each one assembles for you
  16. Practice · 15Context in production: versions, drift, refresh, observability, incidents
  17. Practice · 16The insurance lab: one submission, every context decision
  18. ReferenceGlossary
  19. AboutAbout this site and its sources
Retrieval · 4

Retrieval fundamentals: lexical search, embeddings, hybrid search, reranking

Four ways to find the right passage among many: matching words, comparing meaning as vectors, combining both, and letting a slower scorer reorder the short list.

You will understand
Four ways to find the right passage among many: words, meaning, both, and a second look.
You will see
Where each one fails, on the same query, in the same small corpus.
You will be able to
Say why a filter must run before similarity, and why top-1 is rarely enough.

Case UW-206 is one of this site's own cases. The submission for a roof of 28 years leaves the wind zone empty. It does give the distance to the coast: 0.6 miles. The policy fixture can handle that, but only with two passages. One is the roof rule, R-ROOF-01: older than 25 years in zone C is not eligible. The other is a definition in a different document, DEF-WIND-ZONE: a location within 1 mile of the coast is in zone C. With both, the agent can derive the zone and recommend decline. With only the rule, it cannot apply the rule, because it does not know the zone.

The agent asks search_guidance for "roof age limit and the definition of wind zone C". The corpus has 31 chunks. Which come back first decides whether the window holds the two passages the answer needs, or one of them and four copies of a bulletin.

The previous page treated ranking as one line in the assembler. This page opens that line. It is the most uncertain stage of assembly, because it is the only one that has to guess.

The problem in one sentence#

Retrieval means finding, for a query, the items in a store that are most likely to be relevant. The store here is the 31 chunks of the 20 policy documents (a chunk is a piece of a document stored and retrieved as one unit; RAG explains how documents are cut). The output is a ranked list, and the assembler keeps the first few. That cut is called top-k: keep the k best-ranked items. The example uses k = 5 for passages.

Every retrieval method answers the same question, "how relevant is this chunk to this query?", with a number, and every method is wrong in its own way. There are four common answers.

The oldest answer counts shared words. Lexical search scores a chunk higher the more of the query's words it contains, especially rare words, and especially in short chunks. The standard formula for that is BM25. It gives each query word a weight that is high when few chunks contain the word, adds up those weights for the words the chunk contains, and dampens both repetition (the tenth "roof" adds little) and length (a long chunk contains more words by chance).

This is the whole function in the example, retrieve.py:bm25_score, with the usual constants k1 = 1.2 and b = 0.75:

def bm25_score(query_terms: list[str], doc_terms: list[str], df: dict[str, int], n: int, avg_len: float) -> float:
    """BM25 score of one chunk for one query."""
    score = 0.0
    for t in sorted(set(query_terms)):
        tf = doc_terms.count(t)
        if not tf:
            continue
        idf = math.log(1 + (n - df.get(t, 0) + 0.5) / (df.get(t, 0) + 0.5))
        score += idf * tf * (BM25_K1 + 1) / (tf + BM25_K1 * (1 - BM25_B + BM25_B * len(doc_terms) / avg_len))
    return score

Lexical search is excellent at one thing: exact strings. Ask for R-ROOF-03 and it finds the one chunk that contains that id. Its weakness is the mirror image. A query that uses different words for the same idea scores zero on the chunk that answers it. "Aged roof covering in a storm area" shares no word with "roof older than 25 years in wind zone C" except "roof".

Before AI. This is the inverted index behind full-text search in a relational database or a search engine: a table from each word to the documents that contain it, plus a scoring formula. Nothing about it is new. What is new is that its output goes into a model's window instead of onto a results page, so a wrong top hit costs an answer, not a click.

Comparing meaning: embeddings#

The second answer compares meaning instead of words. An embedding is a list of numbers that stands for the meaning of a text, produced so that texts with similar meaning get similar lists. "Old roof" and "aged roof covering" end up close together even though they share one word. The lists are called vectors, and searching by them is vector search: embed the query, then rank the chunks by how close their vectors are.

Closeness is usually measured by cosine similarity: the cosine of the angle between two vectors. It is 1 when they point the same way, 0 when they are unrelated, and the length of the vectors does not matter.

The example cannot call an embedding model (no model, no network, no key), so it uses a stand-in embedding: a hand-written table maps words to concepts ("aged", "old" and "worn" share the concept AGE; "storm" and "gale" share WIND), and each concept adds +1 or -1 at a hashed position in a vector of 256 numbers. It is not a neural model. It is built so that the differences between methods are visible on 31 chunks, and its scores mean nothing outside this folder.

Embeddings fail where lexical search shines. The stand-in embedding ignores identifiers, so for the query R-ROOF-03 every chunk scores the same, zero, and the ranking falls back to id order:

query: R-ROOF-03
method=vector k=3 filters=on rerank=off
rank  chunk    evidence        score  status     tenant    audience     title
   1  C-01-01  EV-GUIDE-0101  0.0000  current    tenant-a  underwriting UW-GUIDE roof rules / R-ROOF-01 roof age limit i
   2  C-01-02  EV-GUIDE-0102  0.0000  current    tenant-a  underwriting UW-GUIDE roof rules / R-ROOF-02 unknown roof age
   3  C-01-03  EV-GUIDE-0103  0.0000  current    tenant-a  underwriting UW-GUIDE roof rules / R-ROOF-03 conflicting roof

Lexical search puts the right chunk first with a clear margin (score 4.32 against 0.00). Real embedding models are far better than this stand-in, but the pattern is real and documented: one provider's engineering article on retrieval, read for this page, notes that embeddings capture meaning yet can miss exact matches such as an error code, where lexical matching finds the literal string.

If each method fails where the other succeeds, combine them. Hybrid search runs lexical and vector search on the same query and merges the two rankings into one. The merge in the example is reciprocal rank fusion, retrieve.py:rrf: a chunk gets 1 / (60 + rank) from each list that contains it, and the scores are added. Ranks are used instead of raw scores because BM25 scores and cosine similarities live on different scales and cannot be added directly.

def rrf(rankings: list[list[str]], k: int = RRF_K) -> dict[str, float]:
    """Reciprocal rank fusion: an item scores 1 / (k + rank) in every ranking that lists it."""
    out: dict[str, float] = {}
    for ranking in rankings:
        for rank, cid in enumerate(ranking, start=1):
            out[cid] = out.get(cid, 0.0) + 1.0 / (k + rank)
    return out

A chunk that is second in both lists beats one that is first in one list and absent from the other. That is the point: agreement between two different ways of judging relevance is evidence.

A second look: reranking#

The fourth answer accepts that the first ranking is rough and adds a slower, more careful judge for the top of it. Reranking takes a short list, say the first ten, and scores each item again with a method that would be too expensive for the whole store, usually a model that reads query and chunk together. The example's stand-in reranker scores the share of the query's concepts and identifiers that a chunk covers; it is a script, not a model.

The same query, four ways#

Now back to UW-206. Here are the top five for its query with each method, filters on (real output of retrieve.py --case UW-206, shortened to the ids). The two chunks the answer needs are C-01-01 (the rule) and C-05-03 (the wind zone definition).

RankLexicalVectorHybridHybrid + rerank
1C-05-03 definitionC-01-01 ruleC-01-01 ruleC-01-01 rule
2C-01-01 ruleC-11-01 bulletinC-11-01 bulletinC-11-01 bulletin
3C-11-01 bulletinC-15-01 bulletin copyC-15-01 bulletin copyC-15-01 bulletin copy
4C-12-01 bulletin copyC-12-01 bulletin copyC-12-01 bulletin copyC-12-01 bulletin copy
5C-15-01 bulletin copyC-14-01 bulletin copyC-05-03 definitionC-14-01 bulletin copy
Both needed chunks in the top 5yesnoyesno

Three lessons sit in that table, and none of them is "method X is best".

First, top-1 is not enough. Even the lists that are right at rank 1 are right about only one of the two chunks the answer needs. A recommendation that rests on two passages needs a k large enough for both, and a way to measure whether both arrived. That measure is recall at k, the share of the relevant chunks that made the top k; Measuring is its home.

Second, copies crowd out diversity. Bulletin 2026-07 exists in five near-identical versions (case UW-205 is built on them). Every method ranks several copies above the definition, because a copy that is relevant once is relevant five times. Reranking makes it worse here: the stand-in reranker gives all five copies the same high score and pushes the definition out. The fix is not a better scorer but the deduplication stage of the assembler. The example retrieves a pool of ten, removes near-duplicates, and only then cuts at five, so the definition comes back and UW-206 ends in decline, citing both R-ROOF-01 and DEF-WIND-ZONE.

Third, the numbers belong to this corpus. On the example's 14 labelled queries, lexical search with filters has the best recall of the four settings (0.857 at k = 5), because the stand-in embedding is weak and the queries share words with their answers. With a real embedding model and real users' wording, the result can be the opposite. Measure on your own queries.

Try it yourself. Pick a case, a method, filters on or off, reranking on or off, and k.

Setting (14 labelled queries, k = 5)recall@5precision@5MRRQueries with a blocked chunk in the top 5
Lexical, filters off0.8570.2290.7199
Lexical, filters on0.8570.2290.8690
Vector, filters on0.8210.2140.7100
Hybrid, filters on0.8570.2290.7860
Hybrid + rerank, filters on0.8210.2140.8210

All numbers come from the stand-in retrievers of the example (real output of measure.py --retrieval); they show how the methods differ on 31 synthetic chunks, not how good any retriever is. The agent here is a scripted stand-in, so these results show how the mechanism works, not how good any model is.

Diagram: Retrieval for one query · filter first, score two ways, fuse, rerank, deduplicate, cut

Read left to right. The filters decide what may be scored at all. Two scorers judge relevance in different ways, fusion combines their ranks, a reranker takes a second look at a short pool, and deduplication restores diversity before the cut at k. Click a box to see it on case UW-206.

  1. 1 · Who may read it
    1. Metadata filters: Removes chunks the agent may not read or that are out of date, before any score is computed.
  2. 2 · Two scorers
    1. Lexical (BM25): Scores rare shared words, dampened for repetition and length. Strong on exact ids.
    2. Vector (cosine): Compares meaning as vectors. Strong on paraphrase, weak on identifiers.
  3. 3 · Combine and refine
    1. Hybrid (RRF): Adds 1 / (60 + rank) from each list; agreement wins.
    2. Rerank: A slower scorer re-orders a short list. Here a stand-in that counts covered concepts.
  4. 4 · Keep diversity
    1. Deduplicate: Removes near-copies so that different passages reach the top k.
    2. Top-k: Keeps the five best passages for the window.

Filters belong before similarity#

The last column of that table is the one an insurer cares about most. With filters off, nine of the fourteen queries have at least one chunk in the top five that the underwriting agent must not read: another tenant's roof note, a reinsurance treaty memo, or the superseded 2025.4 rule. With filters on, none do.

"Filters on" in the example means something specific: tenant, audience and status are checked before any similarity is computed (retrieve.py:filter_reason, called inside search). The output says what was blocked and where it would have ranked:

blocked before similarity: C-17-01 (other_tenant; would have ranked 1)
blocked before similarity: C-02-01 (superseded; would have ranked 3)
blocked before similarity: C-16-01 (unauthorized; would have ranked 8)

Read the first line twice. Another tenant's roof rule would have been the best match for UW-206. Retrieval measures relevance; it has no idea who may read what. If the filter ran after the top-k cut, that chunk would take a slot, and removing it afterwards would leave four passages instead of five. If the filter were missing, it would reach the window, which is what happened to the first assembler on Assembly. Case UW-201, this site's own, is built to test exactly this.

Production pitfall. Many vector stores apply metadata filters during an approximate search, and the documentation of each store says whether the filter is applied before, during or after the nearest-neighbor step. A post-filter on an approximate index can return fewer than k results, or none, when the filter is selective. Read the store's documentation, then test with a filter that removes most of the corpus. RAG shows the documented shape for pgvector.
For a coding agent. A coding agent retrieves too, usually with tools instead of an index: a file search by name, a text search for an identifier, a read of a whole file. Searching for an exact function name is lexical search, and it is often the right tool for code, because identifiers matter more than meaning. Coding agents and Dynamic context show how an agent combines a repository map with targeted searches.
What this is not. This page is about how retrieval ranks. The interface of the retrieval tool itself (its name, parameters, schema and errors, as the agent sees it) belongs to Tools Engineering. Measuring retrieval quality as a method (golden sets, judges, trials) belongs to Eval Engineering.
The formulas, with their assumptionsBM25, cosine and reciprocal rank fusion, and what each one assumes

BM25. For a query q and a chunk d: score(q, d) = Σ over distinct terms t in q of IDF(t) · tf(t, d) · (k1 + 1) / (tf(t, d) + k1 · (1 − b + b · |d| / avgdl)), with IDF(t) = ln(1 + (N − n(t) + 0.5) / (n(t) + 0.5)). Here tf is the count of t in d, |d| the chunk length in terms, avgdl the average length, N the number of chunks and n(t) the number of chunks that contain t. k1 controls how fast repeated terms saturate; b controls length normalization (b = 0: none). Assumptions: terms are independent, and the tokenization of query and chunk is the same; a change in stemming or stop words changes every score. The index statistics must come from the whole corpus, not from the filtered subset, or a filter changes the scores of the chunks it keeps (the example passes the full-corpus index for this reason).

Cosine similarity. cos(u, v) = u · v / (|u| |v|). With vectors normalized to length 1, it is the dot product. Assumptions: the embedding model was trained so that semantic similarity maps to angle; that holds for the domain and language it was trained on and weakens elsewhere. Query and documents must be embedded with the same model and version; mixing versions in one index silently degrades every search.

Reciprocal rank fusion. RRF(d) = Σ over rankings r of 1 / (k + rank_r(d)), with k = 60 in the example. It needs only ranks, so it combines scores on different scales without calibration. Its constant decides how much the top of each list dominates. It assumes each list is a reasonable ranking; fusing a poor list with a good one still pulls good items down. Weighted variants exist; the example keeps the plain form.

Rerankers. A cross-encoder reads query and chunk together and outputs a relevance score; it is more accurate per pair and too slow for the whole store, which is why it sees only the first-stage pool. The pool size is a real parameter: a relevant chunk outside the pool can never be reranked into the top k.

For a Principal engineer: what to decide, and what to measureRetrieval is a ranking system with a security boundary inside it

Decide the identity of the caller before retrieval. Tenant and role must be inputs of the search, not a check applied to the answer. That keeps unauthorized text out of the window, and it also keeps it out of the logs of the window.

Decide k from the questions, not from habit. If typical answers rest on two or three passages, k = 1 is wrong by design and k = 3 is fragile. Measure recall at several k on labelled queries and pick the smallest k that gives the recall you need, then budget for it (Budgeting).

Keep diversity a separate stage. Rerankers optimize relevance per item, not coverage of the answer. Near-copies, multiple sections of one long document and repeated templates all crowd the top. Deduplication (as in the example) or a diversity-aware selection (maximal marginal relevance is a common name for one) belongs between ranking and the cut.

Version the retriever. The embedding model, the tokenizer of the lexical index, the fusion constant, the reranker and k are part of the context version. Changing any of them changes windows without a code change in the agent. Production shows how to see such drift.

Agentic search is a competing design. Instead of one retrieval per step, the agent issues several searches, reads, and refines. It handles open-ended questions better and costs more calls and less predictability. Dynamic context compares the two.

A checklist for retrieval#

CheckWhy
Tenant, audience and status filters run before similarityA blocked chunk must not take a slot or reach the window
Index statistics come from the whole corpusA filter must not change the scores of the chunks it keeps
Exact identifiers (rule ids, codes) are searchable lexicallyEmbeddings can miss them
k is chosen from how many passages an answer needsTop-1 misses multi-passage answers (UW-206)
Near-duplicates are removed before the cut at kCopies crowd out the passage you need (UW-205)
Recall at k is measured on labelled queries from your domainMethod rankings do not transfer between corpora
The retriever configuration is versionedA change to it changes windows without a code change

In one breath#

Retrieval guesses which chunks are relevant: lexical search matches words and is strong on exact ids, embeddings compare meaning and are strong on paraphrase, hybrid search fuses both rankings, and a reranker takes a slower second look at the top. Each fails somewhere, so measure recall at k on your own labelled queries instead of trusting a method's reputation. And remember that relevance is not permission: filter by tenant, audience and version before similarity, then deduplicate before you cut.

What comes next#

You can now rank 31 chunks for a query. But where did the chunks come from, what metadata did they carry for the filters to use, how does the answer point back to them, and what happens when one of them is out of date? The next page, RAG, treats retrieval-augmented generation as a whole system: chunking, metadata, grounding, citations and freshness.