All chapters
  1. Start hereA model knows only what it can see now
  2. Foundations · 1The context window: tokens, messages and order
  3. Foundations · 2Signals and authority: who is allowed to instruct the model
  4. Foundations · 3The context assembly pipeline: select, rank, deduplicate, pack
  5. Retrieval · 4Retrieval fundamentals: lexical search, embeddings, hybrid search, reranking
  6. Retrieval · 5RAG architecture: chunks, metadata, grounding, citations, freshness
  7. Budget · 6Context budgeting: truncation, dilution, token cost and caching
  8. Budget · 7Compression, summarization and compaction, with loss tests
  9. Agents · 8Coding-agent context: repository maps, instruction files, diffs and tool outputs
  10. Agents · 9Working context, durable memory and checkpoint state
  11. Agents · 10Dynamic context: just-in-time retrieval, selective tool disclosure, delegation
  12. Agents · 11Multi-agent context isolation and scoped handoffs
  13. Trust · 12Context security: injection, poisoning, access controls, tenant boundaries
  14. Trust · 13Measuring context quality: recall, utilization, grounding, ablation
  15. Practice · 14Frameworks and standards: what each one assembles for you
  16. Practice · 15Context in production: versions, drift, refresh, observability, incidents
  17. Practice · 16The insurance lab: one submission, every context decision
  18. ReferenceGlossary
  19. AboutAbout this site and its sources
Retrieval · 5

RAG architecture: chunks, metadata, grounding, citations, freshness

Retrieval-augmented generation as a system: how documents become chunks with metadata, how an answer stays tied to its sources, and how a stale chunk gets into a current answer.

You will understand
Retrieval-augmented generation as a system: chunks, metadata, filters, grounding, citations, freshness.
You will see
How a superseded rule scores as high as the current one, and what stops it.
You will be able to
Check that every citation in an answer points at something that was in the window.

On 1 July 2026 the insurer's policy fixture moved from version 2025.4 to 2026.2. The roof limit of rule R-ROOF-01 in wind zone C dropped from 30 years to 25. The policy team published the new guide. Nobody deleted the old one: auditors need it, and past decisions refer to it.

Case UW-204, this site's own, arrives in August: a roof of 27 years in zone C. Under the current rule that roof is not eligible. Under the old rule it was fine. The agent searches the guidance for "roof age limit in wind zone C", and this is what plain retrieval returns, with no filters (real output of retrieve.py --case UW-204 --method hybrid --no-filters):

rank  chunk    evidence        score  status     tenant    audience     title
   1  C-17-01  EV-NOTE-1701   0.0328  current    tenant-b  underwriting Roof guidance for the tenant B property book / R
   2  C-01-01  EV-GUIDE-0101  0.0320  current    tenant-a  underwriting UW-GUIDE roof rules / R-ROOF-01 roof age limit i
   3  C-02-01  EV-GUIDE-0201  0.0315  superseded tenant-a  underwriting UW-GUIDE roof rules / R-ROOF-01 roof age limit i

The current rule is second. The superseded rule is third, with a score almost equal to the current one. That is not a bug in the retriever. The two chunks say nearly the same thing about the same subject; only a number and a date differ, and relevance scoring does not read dates. In first place sits a roof rule that belongs to a different tenant of the platform.

Retrieval found relevant text. Whether that text is current, whether this agent may read it, and whether the answer will say which of the three it relied on are not questions a ranking can answer. They are the questions of this page.

RAG in one sentence#

RAG, retrieval-augmented generation, means: before the model answers, retrieve relevant passages from a store and put them in the window, so that the answer can rest on them. The Retrieval page covered how passages are ranked. This page covers the system around the ranking: how documents become retrievable pieces, what has to be stored next to each piece, how the answer stays tied to its sources, and how the store stays current.

What this is not. RAG is not memory. A RAG store holds documents that exist independently of the agent (the policy guide); memory holds what the system learned or decided during earlier work. The boundary is on Memory boundary, and persistence in depth is Memory Engineering. The runtime view of RAG inside an agent loop is on Harness Engineering, RAG.

Documents become chunks#

A model cannot take a whole library into its window, and a retriever that returns whole documents wastes the budget on pages that do not matter. So documents are cut into chunks: pieces stored and retrieved as one unit. The act of cutting is chunking.

The example cuts by heading (corpus.py:chunk_documents): each section of a document becomes one chunk. The 20 policy documents become 31 chunks. The roof guide has three sections, one per roof rule, so R-ROOF-01, R-ROOF-02 and R-ROOF-03 are three separate chunks, C-01-01 to C-01-03.

Chunk size is a real trade-off, and both directions fail:

Chunks that are too largeChunks that are too small
A chunk about roofs also carries a paragraph about sprinklers; the window pays for bothThe sentence "not eligible" is separated from the sentence that says which roofs
Fewer, broader chunks match many queries weaklyA chunk loses the heading, version and document it came from
One relevant sentence buys a page of tokensAn answer needs more chunks, so k must grow

Cutting by the document's own structure (headings, rules, clauses) is usually better than cutting every N tokens, because a rule or a clause is the unit a citation should point at. When the structure is poor, overlapping windows of text or a short context line added to each chunk help. One provider's engineering article, read for this page, describes generating such a context line per chunk with a model before indexing, so that a chunk keeps the information of where it fits in its document.

Metadata travels with the chunk#

The superseded rule in UW-204 could only be filtered out because the chunk knows it is superseded. A filter can use only what was stored next to the text. In the example every chunk carries its document's metadata:

FieldC-01-01 (current rule)C-02-01 (old rule)
doc_idDOC-01DOC-02
version2026.22025.4
effective2026-07-012025-01-01
statuscurrentsuperseded
audienceunderwritingunderwriting
tenanttenant-atenant-a
trustinternal_datainternal_data
rule_idsR-ROOF-01R-ROOF-01
evidence_idEV-GUIDE-0101EV-GUIDE-0201

A metadata filter is a condition on these fields that runs before similarity: the right tenant, an audience this agent may read, status current. With those three filters, the retrieval for UW-204 becomes:

blocked before similarity: C-17-01 (other_tenant; would have ranked 1)
blocked before similarity: C-02-01 (superseded; would have ranked 3)
blocked before similarity: C-16-01 (unauthorized; would have ranked 7)

and the current rule is first. The assembler records the superseded chunk in the manifest with the omission reason superseded, so a reviewer can see that the old rule existed, was relevant, and was kept out on purpose.

Before AI. This is a WHERE clause on an indexed table, and it should be designed the same way: the columns you filter on are part of the schema, they are written at ingestion, and a row without them is a data-quality bug. The extra step for RAG is that the filter must be applied inside the vector search, not to its top-k result, or it can leave you with too few rows.

The store: a relational database with a vector column#

Many teams store chunks, metadata and embeddings in the database they already run. pgvector is an extension for PostgreSQL that adds a vector column type, distance operators and approximate indexes. Here is the documented shape for the example's chunks (from the pgvector README read for this page; documented shape; not run here):

CREATE EXTENSION vector;

CREATE TABLE policy_chunk (
  chunk_id     text PRIMARY KEY,
  doc_id       text NOT NULL,
  version      text NOT NULL,
  effective    date NOT NULL,
  status       text NOT NULL,
  audience     text NOT NULL,
  tenant       text NOT NULL,
  rule_ids     text[] NOT NULL,
  body         text NOT NULL,
  embedding    vector(256)
);

CREATE INDEX ON policy_chunk USING hnsw (embedding vector_cosine_ops);

SELECT chunk_id, rule_ids
FROM policy_chunk
WHERE tenant = 'tenant-a' AND audience = 'underwriting' AND status = 'current'
ORDER BY embedding <=> $1
LIMIT 5;

The <=> operator is cosine distance. The index (hnsw) is a vector index: a structure that finds the nearest vectors approximately, without comparing the query with every row. Approximate is the important word, and the README is explicit about the consequence for filters: with an approximate index, filtering is applied after the index is scanned, so a selective WHERE can return fewer rows than the LIMIT asked for. Its example: if 10 percent of rows match the filter and the default search width is 40, about 4 rows come back on average. The README describes iterative index scans (hnsw.iterative_scan) that keep scanning until enough rows match, with settings that cap how far the scan goes.

For the insurer this matters directly. A tenant filter is selective by design. A query that silently returns three chunks instead of five can drop the one rule the case needs. Test retrieval with your real filters, check the count of rows returned, and record it in the manifest.

The README also suggests combining vector search with PostgreSQL full-text search, merged by reciprocal rank fusion or a reranker, which is the hybrid search of the previous page.

Diagram: RAG as a system · what is written at ingestion decides what the window can trust at query time

The left column is ingestion, the right column is one model call. Everything the query side relies on (metadata, versions, stable ids) is written on the ingestion side. Click a box to see it in the example.

  1. Ingestion (when documents change)
    1. Documents: The source texts, each with a version, an effective date, an audience and a tenant.
    2. Chunking: Cuts each document along its own structure so that one rule is one chunk.
    3. Metadata: Tenant, audience, version, effective date, status, trust and rule ids stored next to the text.
    4. Store: Text, metadata and embedding in one row; an approximate index on the vector.
  2. Query time (every model call)
    1. Filter, then rank: Tenant, audience and status conditions applied before or during similarity, then hybrid ranking.
    2. Window: The kept chunks enter the window with version and status in their labels.
    3. Answer with citations: The recommendation names the rule ids it rests on.
    4. Grounding check: Code checks every citation against the manifest of the call.

Grounding and citations#

Retrieval puts passages into the window. Nothing forces the answer to use them. An answer is grounded when every claim in it rests on an item that was in the window. A citation is the pointer from the answer to that item: in the example, a rule id such as R-ROOF-01, which the manifest maps to an evidence id such as EV-GUIDE-0101.

Grounding is checkable, and the check is cheap: for every citation in the recommendation, was the cited rule's text in the window of that call? The example does this in measure.py:grounding. Case UW-008 is built to fail it. The agent declines a roof of 31 years in zone C and cites R-ROOF-01, which is the right answer, but it skipped search_guidance, so the rule's text was never in its window:

case    rec          used/data items token share  grounded
UW-001  decline            4/16            0.276 1/1
UW-008  decline            3/8             0.476 0/1

Same recommendation, same citation. One is grounded (1 of 1 citations was in the window), the other is not (0 of 1). The second agent cited from memory. Today the remembered rule happens to be the current one. After the next policy change, it will cite a limit that no longer exists, and nothing in the answer will show it.

Common confusion. A citation is not proof by itself. A model can produce a plausible rule id that was not in its window, or cite a passage that does not say what the answer claims. The useful check compares citations with the manifest (was it in the window?) and, for higher stakes, the cited text with the claim (does it say that?). Measuring is the home of both measurements.

Providers offer citation features that make the first check structural. The Claude documentation read for this page describes a citations option on document content blocks: the response then contains the exact cited text with its location in the provided document, the API guarantees valid pointers to the provided documents, and the cited text does not count toward output tokens. The same page says the feature cannot be combined with structured outputs. That trade-off matters for this agent, whose recommendation is JSON: you either use the provider's citations and parse a mixed response, or require rule ids in your JSON and check them yourself against the manifest, as the example does.

Freshness#

Freshness is the property that the items in the window are the versions in force at the time of the call. UW-204 broke it with a superseded chunk. Case UW-007, a shared case, breaks it with a decision: a similar roof of 28 years in zone C was approved under 2025.4. Under 2026.2 that roof is not eligible. A past decision is legitimate evidence of how the insurer treated similar risks, and it is wrong evidence once the rule has changed.

The example handles both the same way: every candidate carries the policy version it belongs to, and the freshness stage of the assembler omits anything from a version no longer in force, with the reason superseded and a detail ("decided under 2025.4; the policy in force is 2026.2"). Three operational rules follow:

  1. Store versions, do not overwrite them. Mark the old version superseded at the moment the new one takes effect. Deleting it breaks audits; leaving it unmarked breaks answers.
  2. Filter by effective date, not by ingestion date. A guide can be loaded before it takes effect. The question is which version applies to this submission today.
  3. Re-check derived items. Past decisions, summaries and cached answers made under an old version are stale too, even if their text never mentions the version.
For a coding agent. Freshness bites coding agents as documentation drift: the agent retrieves an older version of an API's documentation, or a README that describes a flag the code removed last month. The fix has the same shape: version the sources, prefer the documentation of the version actually installed (read from the lock file), and make the version visible in the window.
Ingestion is where most RAG bugs are bornA checklist for the pipeline that fills the store

Every property the assembler relies on at query time was written, or not written, at ingestion. In review, walk the ingestion pipeline with these questions.

QuestionFailure if the answer is no
Does every chunk get tenant, audience, version, effective date and status at write time?Filters cannot exclude what they cannot see
Is the status of the old version changed in the same transaction that publishes the new one?A window can hold both, or neither, for a while
Are chunk ids stable across re-ingestion of the same document?Citations and manifests from last week point at nothing
Is the embedding model version stored per row?A model upgrade mixes incompatible vectors in one index
Is text that looks like markup or a delimiter escaped?A document can close a data block early in the window
Are documents from outside the organization labelled external?Untrusted text enters the window as internal data
Is deletion propagated (a withdrawn document leaves the index)?Withdrawn guidance keeps answering

The last row is an incident pattern of its own: a document is withdrawn in the content system, but the index is refreshed only nightly or only on additions. Production covers index refresh.

For a Principal engineer: design trade-offs in a RAG systemStore, filter placement, citation strategy, and when not to use RAG

One store or two. Keeping chunks, metadata and vectors in the transactional database (pgvector) gives you joins, transactions and one access-control model; a dedicated vector store can give better approximate-search performance at scale and adds a second system whose filters and consistency you must verify. Either way, the authorization filter must run inside the search.

Filter placement and recall. With approximate indexes, a selective filter costs recall unless the store scans further. Measure recall at k under your real filter selectivity, not on an unfiltered benchmark.

Citation strategy. Provider-side citations give verified pointers to provided documents and free output tokens for quotes; application-side citations (rule ids in a JSON field) keep structured output and need your own grounding check. The second is easier to log, test and compare across models.

When RAG is the wrong tool. If the rules are few, stable and must be applied exactly, put them in code (as the example's check_eligibility does) and put their ids and text in the window deterministically, without retrieval. RAG is for large, changing corpora where you cannot know in advance which passage a case needs. A deterministic lookup by rule id is a perfectly good assembler stage.

A checklist for a RAG system#

CheckWhy
Chunks follow the document's structure (rule, clause, section)A citation should point at a meaningful unit
Every chunk carries tenant, audience, version, effective date, statusFilters can use only stored metadata
Filters run inside the search, and the number of rows returned is recordedApproximate indexes can return fewer than k
Superseded versions are kept and marked, not deleted or left unmarkedAudits need them; answers must not use them (UW-204)
Derived items (past decisions) carry the version they were made underThey go stale with the rule (UW-007)
Every citation is checked against the manifest of its callA citation from memory is not grounded (UW-008)
The embedding model version is stored per rowMixed versions degrade every search silently

In one breath#

RAG is retrieval plus the system that makes it trustworthy: documents cut along their own structure, every chunk carrying the metadata that filters need, filters applied inside the search, superseded versions kept but marked, and an answer whose citations can be checked against what was actually in the window. Relevance scoring cannot tell a current rule from a stale one or your tenant's text from another's; only metadata can. And a correct citation of a rule that was never in the window is a warning, not a success.

What comes next#

You can now fill a window with current, authorized, cited passages. But every passage costs tokens, and the window is shared with instructions, tool definitions, the submission, tool results and history. The next page, Budgeting, is about dividing a finite window: shares, truncation, why more text can make answers worse, what tokens cost, and what keeps a cached prefix valid.