All chapters
  1. Start hereA model knows only what it can see now
  2. Foundations · 1The context window: tokens, messages and order
  3. Foundations · 2Signals and authority: who is allowed to instruct the model
  4. Foundations · 3The context assembly pipeline: select, rank, deduplicate, pack
  5. Retrieval · 4Retrieval fundamentals: lexical search, embeddings, hybrid search, reranking
  6. Retrieval · 5RAG architecture: chunks, metadata, grounding, citations, freshness
  7. Budget · 6Context budgeting: truncation, dilution, token cost and caching
  8. Budget · 7Compression, summarization and compaction, with loss tests
  9. Agents · 8Coding-agent context: repository maps, instruction files, diffs and tool outputs
  10. Agents · 9Working context, durable memory and checkpoint state
  11. Agents · 10Dynamic context: just-in-time retrieval, selective tool disclosure, delegation
  12. Agents · 11Multi-agent context isolation and scoped handoffs
  13. Trust · 12Context security: injection, poisoning, access controls, tenant boundaries
  14. Trust · 13Measuring context quality: recall, utilization, grounding, ablation
  15. Practice · 14Frameworks and standards: what each one assembles for you
  16. Practice · 15Context in production: versions, drift, refresh, observability, incidents
  17. Practice · 16The insurance lab: one submission, every context decision
  18. ReferenceGlossary
  19. AboutAbout this site and its sources
Trust · 13

Measuring context quality: recall, utilization, grounding, ablation

Four questions with numbers: did retrieval find what was needed, did the model use what it was given, does the answer rest on the sources, and what changes when one item is removed.

You will understand
Four questions with numbers: did retrieval find it, did the model use it, does the answer rest on it, and what carried it.
You will be able to
Compute recall, precision, MRR, utilization and grounding for one window, and read an ablation table.
You will see
Why removing one stage at a time can hide a stage that matters.

The current context version of the underwriting agent, ctx-2026.11, gets all sixteen cases of the example right. The first version, ctx-2026.10, got eight. Between the two, the team changed seven things at once: field selection, filters before similarity, freshness, hybrid retrieval with a reranker, deduplication, budget shares and data labels.

A month later an engineer proposes to remove the reranker, because it adds a second scoring pass to every call. Is that safe? The team reruns the sixteen cases without it and gets sixteen of sixteen again. So the reranker does nothing?

Not quite. Sixteen passes say that no recommendation changed. They do not say whether the right passage now ranks fifth instead of first, whether the window carries more unused text than before, or whether the agent still cites rules it could read. A pass rate is the last number in a chain, and by the time it moves, several numbers before it have already moved. This page measures the chain.

Four questions, four places to measure#

Every recommendation the agent writes comes out of a short chain. Retrieval proposes passages. The assembler puts some of them in the window. The model reads the window and writes an answer with citations. The model recommends; a person decides. A wrong recommendation can start at any link, so each link gets its own question and its own number.

QuestionWhere it is measuredMeasure on this page
Did retrieval find what the case needs?The ranked list, before assemblyrecall@k, precision@k, MRR
Did the model use what it was given?The window against the answercontext utilization
Does the answer rest on what was in the window?The citations against the windowanswer grounding
Which items carried the answer?The answer, with one item removedablation

The first three are cheap and can run on every call or every release. The fourth is an experiment: it reruns the model once per removed item, so it runs on chosen cases.

The agent here is a scripted stand-in, so these results show how the mechanism works, not how good any model is. The stand-in retrievers (BM25, a stand-in embedding and a stand-in reranker) are the ones the Retrieval page describes, and every number below comes from them.

Diagram: Four links, four numbers · where each measure looks (UW-004, UW-206, UW-008)

Read left to right. A recommendation comes out of a chain: a ranked list, a window, an answer with citations. Each measure looks at one link, so a drop shows where it started. Ablation is the experiment that checks the other three.

  1. 1 · The ranked list
    1. Recall at k: Share of the labelled relevant chunks that reached the top k.
    2. Precision at k: Share of the top k that is relevant; near-copies count once.
    3. Reciprocal rank: 1 / rank of the first relevant chunk; averaged over queries it is the mean reciprocal rank.
  2. 2 · The window
    1. Utilization: Share of the window's data items and tokens that the answer rested on.
  3. 3 · The answer
    1. Grounding: Share of the citations whose source text was readable in the window of that call.
  4. 4 · The experiment
    1. Ablation: Remove one item or one stage, rerun, compare the answer with the base answer.

Did retrieval find it: recall and precision at k#

Start with the plainest question. A person has marked, for a query, which chunks answer it. Those are the labelled relevant chunks. For case UW-206 the query is "roof age limit and the definition of wind zone C", and two chunks answer it: C-01-01 (the roof rule R-ROOF-01) and C-05-03 (the definition of wind zone C, rule id DEF-WIND-ZONE).

Retrieval recall at k is the share of the relevant chunks that appear in the first k results. If both relevant chunks are in the top 5, recall@5 is 1.0. If only one is, it is 0.5. Recall answers "did the window have a chance?", because a chunk that never reaches the top k never reaches the window.

Retrieval precision at k is the share of the first k results that are relevant. Two relevant chunks in a top 5 give 0.4. Precision answers "how much of the passage budget went to text the case does not need?"

MRR, mean reciprocal rank, looks only at the first relevant result. For one query, the reciprocal rank is 1 divided by the rank of the first relevant chunk: 1.0 when it is first, 0.5 when it is second, 0.25 when it is fourth. MRR is the mean of that over all labelled queries. It answers "how high does the first useful passage sit?", which matters when the budget is tight or k is small.

One rule of the example changes how these are counted: near-copies count once. Bulletin 2026-07 exists in five near-identical copies (case UW-205), and five copies in a top 5 are one hit, not five. Otherwise a ranking full of copies would score as excellent precision.

Here are the three measures for the UW-206 query at k = 1 to 5, under two retriever settings. The script was written for this page; it calls retrieve.py:search and measure.py:retrieval_metrics. A successful run shows that the same query can have perfect MRR and half its evidence missing.

query: roof age limit and the definition of wind zone C   relevant: C-01-01 C-05-03
lexical, filters on: top 5 = C-05-03 C-01-01 C-11-01 C-12-01 C-15-01
  k=1  recall 0.50  precision 1.00  reciprocal rank 1.00
  k=2  recall 1.00  precision 1.00  reciprocal rank 1.00
  k=3  recall 1.00  precision 0.67  reciprocal rank 1.00
  k=4  recall 1.00  precision 0.50  reciprocal rank 1.00
  k=5  recall 1.00  precision 0.40  reciprocal rank 1.00
hybrid + rerank, filters on: top 5 = C-01-01 C-11-01 C-15-01 C-12-01 C-14-01
  k=1  recall 0.50  precision 1.00  reciprocal rank 1.00
  k=2  recall 0.50  precision 0.50  reciprocal rank 1.00
  k=3  recall 0.50  precision 0.33  reciprocal rank 1.00
  k=4  recall 0.50  precision 0.25  reciprocal rank 1.00
  k=5  recall 0.50  precision 0.20  reciprocal rank 1.00

Read the second block. The reciprocal rank is 1.00 at every k: the first result is the roof rule, which is relevant. Recall never passes 0.50: the wind zone definition is not in the top 5, because the bulletin copies took the slots. A dashboard that tracked only MRR would show this query as perfect. The window would still lack half of what the answer needs. In the assembled window the deduplication stage repairs it (the copies go, the definition comes back), which is why UW-206 still ends in decline citing both R-ROOF-01 and DEF-WIND-ZONE.

Precision falls as k grows, by arithmetic: two relevant chunks can never fill five slots. Low precision is not a failure in itself. It is a cost, paid in tokens, and recall is the property you cannot do without.

Over all 14 labelled queries of the example, measure.py --retrieval prints one row per setting. The last column counts queries whose top 5 held a chunk the agent may not read (another tenant's, another audience's, or a superseded one):

14 labelled queries, k=5 (stand-in retrievers)
method   filters rerank recall@k  prec@k    MRR  not-allowed-in-top-k
lexical  off     off       0.857   0.229  0.719  9 queries
lexical  off     on        0.821   0.214  0.725  9 queries
lexical  on      off       0.857   0.229  0.869  0 queries
lexical  on      on        0.821   0.214  0.869  0 queries
vector   off     off       0.821   0.214  0.589  8 queries
vector   off     on        0.821   0.214  0.672  8 queries
vector   on      off       0.821   0.214  0.710  0 queries
vector   on      on        0.821   0.214  0.798  0 queries
hybrid   off     off       0.821   0.214  0.676  9 queries
hybrid   off     on        0.821   0.214  0.711  9 queries
hybrid   on      off       0.857   0.229  0.786  0 queries
hybrid   on      on        0.821   0.214  0.821  0 queries

Now the reranker question from the opening has an answer in numbers. With hybrid retrieval and filters on, the reranker raises MRR from 0.786 to 0.821 and lowers recall@5 from 0.857 to 0.821. It puts the first useful passage higher and, on one query, pushes a second needed passage out. Neither effect showed in the pass rate. Whether to keep it is a decision about which of the two numbers your cases depend on, and the numbers belong to this corpus and this stand-in, not to rerankers in general.

Change the method, the filters, the reranker and k yourself:

Case, setting (k = 5)Top 5recall@5precision@5Reciprocal rank
UW-206, lexical, filters onC-05-03, C-01-01, C-11-01, C-12-01, C-15-011.000.401.00
UW-206, hybrid, filters, rerankC-01-01, C-11-01, C-15-01, C-12-01, C-14-010.500.201.00
UW-206, hybrid, filters, rerank, drop near-copies(the definition returns)1.000.401.00
UW-201, lexical, no filters (first version)C-17-01, C-02-01, C-16-01, C-01-01, C-11-011.000.200.25
UW-201, hybrid, filters, rerank (current)C-01-01, C-11-01, C-12-01, C-15-01, C-13-011.000.201.00

In the first-version row of UW-201, three of the five slots hold chunks the agent may not read, and recall is still a perfect 1.00. Recall says the right chunk arrived; it says nothing about what arrived with it. That is why the example counts not-allowed chunks as a separate number.

Under the hood. To compute recall on real traffic you need the ranked list each call received, not only the passages that reached the window. The OpenTelemetry semantic conventions for generative AI, read for this page (status: Development), define a gen_ai.retrieval span with gen_ai.retrieval.top_k and an opt-in attribute gen_ai.retrieval.documents whose entries should carry an id and a score. Recorded with stable chunk ids, that is enough to join a production ranking with your labels offline. The conventions read here define no attribute for chunks a filter removed; Production shows where that record goes instead.
What this is not. This page defines the measures of context quality and runs them on one example. How to build the labelled set, how many cases are enough, how to grade with a model as judge, and how to turn a number into a release gate is measurement method, and it belongs to Eval Engineering; its chapters on golden datasets and judges are the place to start.

Did the model use it: context utilization#

Retrieval measures the list. The next question is about the window. Context utilization is the share of the data in the window that the answer actually rested on, counted in items and in tokens. If a window carries 16 data items and the recommendation used 3, item utilization is 3 of 16. Instructions, tool definitions and the new turn are not counted; they are there for every call.

"Used" needs a definition you can compute. In the example, the stand-in agent reports which items its decision read facts from (agent.py:decide returns used), and measure.py:utilization divides. With a real model you need a proxy: the items the answer cites, the items an attribution method points at, or the items whose removal changes the answer, which is the ablation of the next section.

For UW-004, the roof conflict of SUB-1004, a script written for this page prints the items:

UW-004: escalate cites ['R-ROOF-03']
  data items in the window: 16; used: 3 -> EV-SUB-1004-ROOF EV-ROOF-1004-A1 EV-GUIDE-0103
  item share 0.188  token share 0.224
  grounded ['R-ROOF-03']  ungrounded []  share 1.0

Three items carried the recommendation: the application's roof age (9 years), the provider's roof assessment (22 years) and the text of R-ROOF-03, which says that a gap of more than 5 years goes to a senior underwriter. The other 13 data items (the exposure, identity, size, protection and losses field groups, four more passages, a past decision, three earlier turns) were read and not needed for this answer.

Over all sixteen cases (measure.py --cases), token share ranges from 0.176 (UW-006) to 0.476 (UW-008):

case    rec          used/data items token share  grounded
UW-001  decline            4/16            0.276 1/1
UW-002  approve            6/16            0.283 0/0
UW-003  request_info       3/15            0.246 1/1
UW-004  escalate           3/16            0.224 1/1
UW-005  decline            4/16            0.287 1/1
UW-006  request_info       3/18            0.176 0/0
UW-007  decline            4/15            0.298 1/1
UW-008  decline            3/8             0.476 0/1
UW-009  approve            6/16            0.278 0/0
UW-010  escalate           3/16            0.224 1/1
UW-201  decline            4/15            0.297 1/1
UW-202  decline            4/16            0.390 1/1
UW-203  escalate           3/13            0.218 1/1
UW-204  decline            4/16            0.276 1/1
UW-205  decline            4/16            0.276 1/1
UW-206  decline            5/16            0.377 2/2
Common confusion. Low utilization is not waste by definition. The exposure field group was not used in UW-004, but it would have been needed if the roof ages had agreed and the roof were old: the window has to hold what the answer might need before the model knows which branch it is on. Utilization is a trend to watch, not a score to maximize. A falling token share after a change says the window grew with text nobody uses; a share near 1.0 says the window holds nothing in reserve. The highest share above, UW-008, is high for a bad reason: its agent skipped retrieval, so its window is small.

Does the answer rest on it: answer grounding#

RAG defined grounding as a property: every claim of an answer rests on an item that was in the window. Here it becomes a number. Answer grounding is the share of the answer's citations whose source text was readable in the window of that call. In the example a citation is a rule id, and measure.py:grounding checks whether that rule's text appears in the window, not counting a passage the window itself labels as superseded.

The same script, for UW-008:

UW-008: decline cites ['R-ROOF-01']
  data items in the window: 8; used: 3 -> EV-SUB-1001-ROOF EV-SUB-1001-EXPOSURE EV-ROOF-1001-A1
  item share 0.375  token share 0.476
  grounded []  ungrounded ['R-ROOF-01']  share 0.0

The recommendation is right: a roof of 31 years in zone C is not eligible under R-ROOF-01, and the agent declines. The grounding is 0 of 1. The agent skipped search_guidance, the rule's text was never in its window, and the citation came from what the stand-in "remembers". It looks identical to the grounded answer of UW-001. Only this measurement tells them apart, and it will be the first number to warn you when the remembered rule and the rule in force stop agreeing.

Grounding as computed here is the cheap half of the check: was the cited source in the window? The expensive half, does the cited text support the claim, needs a reader of both texts, usually a model as judge, and that is method for Eval Engineering.

Which items carried it: ablation#

Utilization relies on the agent's own account of what it used. Ablation does not. It removes one item from the window, reruns the model on the rest, and compares the answer with the original. An item whose removal changes the recommendation or its citations is load-bearing for that answer. The word comes from experiments where a part is removed to learn what it did.

measure.py --case UW-004 --ablation removes each of the 22 included items except the system instruction, one at a time:

UW-004: base escalate ['R-ROOF-03']
  without TOOLDEF-get_roof_assessment      -> escalate     ["R-ROOF-03"]                same
  without TOOLDEF-search_guidance          -> escalate     ["R-ROOF-03"]                same
  without TOOLDEF-find_similar_decisions   -> escalate     ["R-ROOF-03"]                same
  without TOOLDEF-check_eligibility        -> escalate     ["R-ROOF-03"]                same
  without TOOLDEF-record_recommendation    -> escalate     ["R-ROOF-03"]                same
  without EV-SUB-1004-ROOF                 -> approve      []                           CHANGED
  without EV-SUB-1004-EXPOSURE             -> escalate     ["R-ROOF-03"]                same
  without EV-SUB-1004-IDENTITY             -> escalate     ["R-ROOF-03"]                same
  without EV-SUB-1004-SIZE                 -> escalate     ["R-ROOF-03"]                same
  without EV-SUB-1004-PROTECTION           -> escalate     ["R-ROOF-03"]                same
  without EV-SUB-1004-LOSSES               -> escalate     ["R-ROOF-03"]                same
  without EV-ROOF-1004-A1                  -> approve      []                           CHANGED
  without EV-PROC-0801                     -> escalate     ["R-ROOF-03"]                same
  without EV-GUIDE-0103                    -> approve      []                           CHANGED
  without EV-GUIDE-0102                    -> escalate     ["R-ROOF-03"]                same
  without EV-PROC-0802                     -> escalate     ["R-ROOF-03"]                same
  without EV-GUIDE-0101                    -> escalate     ["R-ROOF-03"]                same
  without EV-DEC-2026-1122                 -> escalate     ["R-ROOF-03"]                same
  without HIST-1004-ROOF-A1                -> escalate     ["R-ROOF-03"]                same
  without HIST-1004-SEARCH                 -> escalate     ["R-ROOF-03"]                same
  without HIST-1004-SIMILAR                -> escalate     ["R-ROOF-03"]                same
  without TURN-1004                        -> escalate     ["R-ROOF-03"]                same

Three removals change the answer, and they are the same three items utilization named. That agreement is a useful check in itself: when the agent's account of what it used and the experiment disagree, one of them is wrong.

The direction of the change matters more than the count. Each of the three removals turns escalate into approve with no citation. Without the provider's 22 years, the window shows a roof of 9 years and no conflict. Without the application's 9 years, there is no conflict either. Without the text of R-ROOF-03, the agent sees two roof ages and no rule that says what a gap means. In every case the agent recommends cover with no reason to stop, and nothing in the answer shows that a piece was missing. That is the failure the manifest exists to catch: an item lost to the budget, a filter or a retriever change produces a confident answer, not an error.

Try every removal for any case:

CaseBase recommendationItems whose removal changes itChanges to
UW-004escalate, R-ROOF-03EV-SUB-1004-ROOF, EV-ROOF-1004-A1, EV-GUIDE-0103 (3 of 22)approve, no citation
UW-001decline, R-ROOF-01EV-SUB-1001-EXPOSURE · EV-GUIDE-0101request_info (no zone) · approve (no rule)
UW-206decline, R-ROOF-01, DEF-WIND-ZONEEV-SUB-1206-EXPOSURE · EV-GUIDE-0101 · EV-REF-0503request_info · approve · request_info
Before AI. Ablation is the experiment a backend engineer runs with a feature flag: turn one thing off, keep everything else equal, compare. It is also how you find the one index a query plan depends on, by dropping it in a copy of the database. The comparison holds for the method. It stops at the subject: a real model is not deterministic, so one rerun per removal is not enough, and an item can matter only in combination with another, which a one-at-a-time removal will not reveal.

Removing a stage instead of an item#

Assembly left an open question. The current version changed several stages at once, so the comparison of ctx-2026.10 with ctx-2026.11 cannot say which stage was necessary. The same experiment answers it one level up: remove one stage of the assembler, keep the others, rerun all sixteen cases.

The example makes that possible because a context version is a dictionary of settings, and assemble.py:config(version, **overrides) replaces any of them. run.py:run_case(case, version, **overrides) passes the overrides through. The script below, written for this page, turns off one stage at a time on top of ctx-2026.11. A successful run shows which stages, removed alone, change a recommendation.

from run import run_case
from scenario import ALL_CASES, load_case

REMOVALS = [
    ("none (ctx-2026.11 as is)", {}),
    ("field selection", {"select_fields": False}),
    ("authorization", {"authorize": False, "filters_before_similarity": False}),
    ("freshness", {"freshness": False}),
    ("filters before similarity (filter after)", {"filters_before_similarity": False}),
    ("deduplication", {"dedupe": False}),
    ("reranking", {"rerank": False}),
    ("hybrid (lexical only)", {"method": "lexical"}),
    ("budget shares (one flat pool)", {"shares": None}),
    ("data labels (plain text)", {"mode": "plain"}),
]

cases = [load_case(c) for c in ALL_CASES]
print(f"{'stage removed':<42} match  cases that change")
for name, overrides in REMOVALS:
    failed = []
    for case in cases:
        r = run_case(case, "ctx-2026.11", **overrides)
        if not r["passed"]:
            failed.append(f"{r['case_id']}:{r['recommendation']['recommendation']}")
    print(f"{name:<42} {16 - len(failed):>2}/16  {' '.join(failed) or '-'}")
stage removed                              match  cases that change
none (ctx-2026.11 as is)                   16/16  -
field selection                            16/16  -
authorization                              16/16  -
freshness                                  14/16  UW-005:approve UW-205:approve
filters before similarity (filter after)   16/16  -
deduplication                              13/16  UW-005:approve UW-205:approve UW-206:request_info
reranking                                  16/16  -
hybrid (lexical only)                      16/16  -
budget shares (one flat pool)              15/16  UW-202:approve
data labels (plain text)                   15/16  UW-005:approve

Two results in this table are surprising, and both are lessons about ablation itself.

Authorization looks useless, and is not. Removing the tenant and audience filters changes no recommendation. The reason is that a second layer covers the same failure: in labelled mode, every passage carries its tenant and status, and the stand-in agent sets aside a passage of another tenant. Remove both layers together and the result is different:

stages removed                             match  cases that change
authorization + data labels                 8/16  UW-001:approve UW-005:approve UW-007:approve UW-201:approve UW-202:approve UW-204:approve UW-205:approve UW-206:approve
freshness + data labels                    14/16  UW-005:approve UW-205:approve

Eight cases fail, the same eight the first version failed. A one-at-a-time ablation measures what a stage adds given all the others, so two layers that protect against the same failure each look unnecessary. Removing either on the strength of that table would leave the system one layer from the first version's failures. For security layers this is the normal case: defense in depth is designed to make each layer look redundant.

Freshness fails through another stage. Without the freshness stage, UW-005 turns into approve. The manifest of that run says why (output of a second script written for this page):

UW-005 current        passages in: EV-BULL-1101 P-NOTE-17 EV-GUIDE-0101 EV-REF-0901 EV-REF-0503
UW-005 freshness off  passages in: EV-BULL-1101 P-NOTE-17 EV-GUIDE-0201 EV-REF-0901 EV-REF-0503
UW-005 freshness off  rule chunks omitted as duplicate: EV-GUIDE-0101 (near-duplicate of EV-GUIDE-0201)

The superseded 2025.4 rule ranked above the current one, and the deduplication stage then removed the current rule as a near-copy of the old one. The window kept the stale rule, which its label marks as superseded, so the agent set it aside and had no roof limit to apply. Removing one stage changed what another stage did. The manifest made that visible in one line, which is the argument of this site in miniature: without the record of what was omitted and why, this would read as "the model approved a 29-year roof".

For a coding agent. The same two experiments work on a coding agent's context. Item ablation: remove one instruction file section or one file read from the session and rerun the task; the sections whose removal breaks the tests are the ones worth their tokens. Stage ablation: turn off clearing of old tool results, or the repository map, and compare test outcomes across a set of tasks. Coding agents shows the fixture where the token side of that comparison is visible.
The measures, defined with their assumptionsWhat each formula assumes, and where it misleads

For a query q with labelled relevant set R (near-copies mapped to one canonical id) and ranking L:

MeasureDefinitionAssumesMisleads when
recall@ksize of (R ∩ top-k of L) / size of RR is complete: every chunk that answers q is labelledThe labels miss a valid answer; a new document is added and not labelled
precision@kdistinct relevant ids in top-k / kk is the number of slots the window actually givesR has fewer than k items: the maximum is size of R / k, not 1
reciprocal rank1 / rank of the first relevant id; 0 if noneOne relevant chunk is enoughThe answer needs several chunks (UW-206): RR is 1.0 with half the evidence missing
MRRmean of reciprocal rank over queriesQueries are equally importantA few critical queries are averaged away by many easy ones
utilization (items, tokens)used data items / data items; their tokens / data tokens"Used" is observableWith a real model "used" is a proxy (citations, attribution, ablation) and inherits its errors
groundingcitations whose source text was in the window / citationsCitations are structured ids that map to itemsFree-text answers: a claim without a citation is invisible to this measure
ablationanswer without item i compared with the base answerDeterminism, and independence between itemsSampling noise reads as an effect; two redundant items each read as unnecessary

Two counting choices in the example are worth copying. Near-copies count once, so duplicates cannot inflate precision. And items the agent may not read are counted separately ("not-allowed in top k"), because a ranking can have perfect recall and still put another tenant's chunk in the window.

With a real model, ablation needs repeated trials per removal and a comparison of distributions, not of single answers: UW-010 gives escalate in 15 of 20 trials on an unchanged window, so a single rerun that differs proves nothing. The number of trials and the statistics belong to Eval Engineering.

For a Principal engineer: what to measure where, and what each costsOnline, offline and on demand; layered defenses; labels as an asset

Three tiers. Grounding and utilization by tokens can be computed online, on every call, from the manifest and the answer, at no model cost. Retrieval recall, precision and MRR need labelled queries, so they run offline on every change to the corpus, the chunking, the retriever or the context version. Ablation costs one model call per removed item per trial, so it runs on demand: on a fixed set of cases before a release that changes the assembler, and on the window of an incident.

Stage ablation finds marginal value, not necessity. When two layers guard the same failure (labels and authorization; filters and the model's instruction to ignore other tenants), one-at-a-time removal shows each as unnecessary. Test such layers in pairs, and do not delete a security layer because a single-stage ablation found no effect. The interaction in the opposite direction exists too: removing freshness changed what deduplication removed. Review stage ablations with the manifests, not only the pass counts.

The labelled set is an asset with an owner. Recall is only as good as its labels. When the policy changes (2025.4 to 2026.2), the relevant chunks of a query change, and stale labels reward the stale chunk. Version the labels with the corpus.

Measure at the right link. A pass rate that stays flat while recall falls is the typical shape of a regression that has not reached the answer yet. Track each link's number separately so the drop is visible before an underwriter sees a wrong recommendation.

A checklist for measuring context#

CheckWhy
Every case's relevant chunks are labelled, with near-copies mapped to one idRecall and precision need a ground truth that duplicates cannot inflate
recall@k is reported with k equal to the slots the window givesA recall at 10 says nothing about a window that keeps 5
Multi-passage cases are measured by recall, not only MRRMRR is perfect with half the evidence missing (UW-206)
Chunks the agent may not read are counted separatelyRecall does not see what arrived with the right chunk (UW-201)
Grounding is computed from the manifest and the citations of every callA correct citation from memory is not grounded (UW-008)
Ablation is run on chosen cases before an assembler changeShows which items carry the answer, and what a lost item does
Layered defenses are ablated in pairsOne-at-a-time removal makes each layer look redundant
Every number names its retriever, its corpus version and its context versionA number without its configuration cannot be compared

In one breath#

A pass rate is the last link of a chain, so measure each link: recall and precision at k for whether retrieval found what the case needs, MRR for how high it sits, utilization for how much of the window the answer used, grounding for whether every citation was in the window, and ablation for which items actually carried the answer. Run the cheap measures on every call, the labelled ones on every change, and ablation on chosen cases. And remember what ablation teaches about itself: two layers that guard the same failure each look unnecessary when you remove them one at a time.

What comes next#

You can now say, with numbers, whether a window had what it needed and whether the answer rested on it. Many teams do not write their assembler from scratch: they take retrieval from a library, caching and token counting from a provider, and a whole loop from an agent SDK. The next page, Frameworks, reads each of them on its own documentation and says which step of building the window it takes over, and which steps, including the manifest these measures need, stay in your code.