All chapters
  1. Start hereA model knows only what it can see now
  2. Foundations · 1The context window: tokens, messages and order
  3. Foundations · 2Signals and authority: who is allowed to instruct the model
  4. Foundations · 3The context assembly pipeline: select, rank, deduplicate, pack
  5. Retrieval · 4Retrieval fundamentals: lexical search, embeddings, hybrid search, reranking
  6. Retrieval · 5RAG architecture: chunks, metadata, grounding, citations, freshness
  7. Budget · 6Context budgeting: truncation, dilution, token cost and caching
  8. Budget · 7Compression, summarization and compaction, with loss tests
  9. Agents · 8Coding-agent context: repository maps, instruction files, diffs and tool outputs
  10. Agents · 9Working context, durable memory and checkpoint state
  11. Agents · 10Dynamic context: just-in-time retrieval, selective tool disclosure, delegation
  12. Agents · 11Multi-agent context isolation and scoped handoffs
  13. Trust · 12Context security: injection, poisoning, access controls, tenant boundaries
  14. Trust · 13Measuring context quality: recall, utilization, grounding, ablation
  15. Practice · 14Frameworks and standards: what each one assembles for you
  16. Practice · 15Context in production: versions, drift, refresh, observability, incidents
  17. Practice · 16The insurance lab: one submission, every context decision
  18. ReferenceGlossary
  19. AboutAbout this site and its sources
Practice · 15

Context in production: versions, drift, refresh, observability, incidents

A window is built from parts that change on different days. How to version them, notice drift, refresh an index, record what each call saw, and reconstruct a bad answer from its context.

You will understand
Why a window is built from parts that change on different days, and how to version them as one.
You will be able to
Decide what to log per call, and explain a wrong recommendation from its manifest.
You will see
A retriever change that altered 11 of 16 windows and no recommendation.

In October the underwriting agent ran with its first context version, ctx-2026.10. One of its runs was case UW-201, one of this site's own cases: submission SUB-1201, a masonry warehouse with a roof of 27 years in wind zone C. Under the policy in force, 2026.2, rule R-ROOF-01 says a roof older than 25 years in zone C is not eligible. The agent recommended approve. The model recommends; a person decides, and the underwriter decided. But in November a reviewer asks the question every production system eventually gets: why did the agent recommend cover for a roof the guidance excludes?

By then a lot has changed. The context version is ctx-2026.11. The index was refreshed. Run the same submission today and the agent recommends decline. That answer is useless to the reviewer. The question is about the window of October, and the window of October no longer exists.

Unless it was recorded, or can be rebuilt. This page is about the window over months: what changes, how to give the configuration a version, how to notice drift, what to log on every call, and how to answer the reviewer from a record instead of a guess.

A window is built from parts that change on different days#

Look at what goes into one window of the underwriting agent and ask who changes each part, and when.

PartWhat changes itHow often (illustrative)
System instruction and tool definitionsA release of the agentWeeks
The policy fixture (UW-GUIDE)The policy team; 2025.4 became 2026.2 on 2026-07-01Months
The index of the 20 policy documentsAn ingestion job: new bulletins, withdrawn notes, a superseded versionDays
The 3,000 past decisionsEvery underwriter decision adds oneHours
Retriever settings: method, reranker, kThe team that owns the assemblerWeeks
The embedding model and the tokenizerA provider or a platform teamMonths
Budget shares and filtersThe team that owns the assemblerWeeks
The model itselfThe provider or a configuration switchMonths

The application code of the agent can stay the same through all of these changes. The windows it builds do not. This is the production problem of context engineering: the input of every model call is a function of many versioned things, and the people who change them are not the people who own the agent.

Before AI. A relational database has the same shape. The query text in your code never changes, but the statistics the optimizer uses are refreshed by a background job, an index is added by another team, and one morning the same query takes a different plan. Good teams record the plan with the slow query. The comparison holds up to the plan: a database plan changes speed, while a changed window changes the answer, and nobody gets an error.

A context version#

A context version is a name for one complete assembly configuration: every setting that decides which items enter a window and how. Two windows built for the same inputs under the same context version must be the same; if a setting changes, the name changes.

The example has two:

Settingctx-2026.10 (first)ctx-2026.11 (current)
Field selectionall 15 groupsthe 6 requested groups
Authorize, freshnessoffon, filters before similarity
Retrievallexical (BM25), no rerankinghybrid (RRF) + stand-in reranker
DeduplicationoffJaccard ≥ 0.8
Budget2,800 tokens, one flat pool2,800 tokens in four shares
Data labelsplainlabelled <data ...> blocks
Cases matching the expected recommendation8 of 1616 of 16

In the code a context version is a dictionary (assemble.py:CONTEXT_VERSIONS), and every manifest carries its name in context_version, next to policy_version. That pairing matters: the policy version says which rules are in force, the context version says how the window was chosen. A wrong answer can come from either, and you need both to tell which.

What belongs inside a context version is everything that changes a window without changing the inputs: retriever method and its parameters, the embedding model and its version, the reranker, k, the deduplication threshold, the budget and its shares, the label format, the filters. What stays outside, recorded separately in the manifest, is what changes with the inputs: the policy version, the index snapshot, the submission.

For a coding agent. A coding agent's context version is its instruction files, its tool list and the harness settings that load and clear context. A change to AGENTS.md changes every later window in that repository, exactly as a change of budget shares does here. Treat it as a reviewed change, and when an agent's behavior shifts, check first whether the instruction file changed. Coding agents shows what those files put in the window.

Drift: the window changes while nobody changes the agent#

Context drift is a change in the windows an agent receives that comes from a change in its parts, not from a change in its inputs or its code. Three kinds matter in the example.

The policy version changes. On 2026-07-01 the roof limit fell from 30 to 25 years. Everything derived from the old version became stale at that moment: the 2025.4 chunks (case UW-204) and the past decisions made under 2025.4 (case UW-007, a similar roof approved under the old rule). Of the 3,000 past decisions in the example, 2,347 were made under 2025.4. The freshness stage keeps them out with the omission reason superseded; without it, the most similar past case for UW-007 would argue for approval.

The index is refreshed. Index refresh is the job that brings the retrieval store up to date with its sources: new documents chunked and embedded, changed ones replaced, withdrawn ones removed, superseded ones marked. Each refresh can change what ranks in the top k for every query. A refresh that adds documents but never removes or re-marks them is the common failure: a withdrawn bulletin keeps answering, or a superseded rule stays current because its status was not updated in the same job that published the new one (RAG lists the ingestion rules).

The retriever or the embedding model changes. A new embedding model ranks the same chunks differently. The example cannot swap in a real model, but it can swap the method. The script below, written for this page, runs all sixteen cases under three settings and counts the omission reasons in their manifests; the third setting is ctx-2026.11 with the retriever switched to vector search only, as a stand-in for an embedding change. A successful run shows drift that the pass rate cannot see.

omitted items per reason, all 16 cases (final attempt)
reason                            ctx-2026.10                ctx-2026.11  ctx-2026.11 + vector only
not_requested                               0                        144                        144
unauthorized                                0                          8                          6
other_tenant                                0                         11                         11
superseded                                  0                         80                         78
duplicate                                   0                         55                         60
below_rank_cutoff                         105                         20                         15
over_budget                                11                          6                          6
ctx-2026.10                mean tokens  2111.6   match 8/16
ctx-2026.11                mean tokens  2254.5   match 16/16
ctx-2026.11 + vector only  mean tokens  2246.3   match 16/16

ctx-2026.11 against the same version with the retriever swapped (vector only)
  UW-001: +1 -1 items  decline -> decline
  UW-002: +2 -2 items  approve -> approve
  UW-003: +1 -1 items  request_info -> request_info
  UW-005: +1 -1 items  decline -> decline
  UW-007: +1 -1 items  decline -> decline
  UW-009: +1 -1 items  approve -> approve
  UW-201: +1 -1 items  decline -> decline
  UW-202: +1 -1 items  decline -> decline
  UW-204: +1 -1 items  decline -> decline
  UW-205: +1 -1 items  decline -> decline
  UW-206: +1 -1 items  decline -> decline
windows changed: 11/16; recommendations changed: 0/16

Two lessons. First, the retriever change altered the window of 11 of 16 cases and no recommendation. Every answer-level check stays green. The next policy change or the next unusual submission may be the one where the swapped passage was the needed one, and the Measuring page showed how quiet that failure is: a missing rule produces a confident approve, not an error. Drift is visible in the manifests before it is visible in the answers.

Second, the distribution of omission reasons is a fingerprint of a context version. ctx-2026.10 omits almost everything as below_rank_cutoff and over_budget; ctx-2026.11 omits mostly by selection, freshness and deduplication. A dashboard of omission reasons per day, per context version, shows a broken filter (the superseded count drops to zero), a growing tool output (over_budget rises) or a flood of copies (duplicate rises) on the day it happens.

Diagram: From changing parts to an explained answer · versions, record, replay (UW-201)

Read left to right. Parts change on different days; a context version names the configuration; every model call writes a manifest joined to its span; an incident is answered by rebuilding the window from the record, not by rerunning today's configuration.

  1. 1 · Parts that change
    1. Policy version: The rules in force; everything made under the old version becomes stale on the effective date.
    2. Index refresh: New, changed, withdrawn and superseded chunks reach the store.
    3. Retriever and embedding: A new method or embedding model reorders the same chunks.
  2. 2 · Name the configuration
    1. Context version: One name for every setting that decides which items enter a window.
  3. 3 · Record every call
    1. Manifest: Ids, versions, included items and omitted items with reasons, for one model call.
    2. Span: The model call in the run's trace, with the attributes of the OpenTelemetry GenAI conventions.
  4. 4 · Explain later
    1. Replay: Rebuild the window of a past call from its recorded inputs and versions.
    2. Incident analysis: Read the rebuilt window, find what misled the answer, name the stage that let it in.

What to log per call#

Context observability is the ability to answer, for any past model call, what its window contained, what was left out, and why, from records kept at the time. It needs one record per model call, and the manifest of this site is designed to be that record.

The manifest of UW-004 carries the identifiers that place it, the versions that explain it, and the two lists:

FieldValue in UW-004Why it is logged
submission_idSUB-1004The business record the underwriter sees
run_id, attempt_number, steprun-7f3a, 1, recommendWhich model call of which execution
trace_id, span_id0c17c0046d4371b42858072814d7b4ca, 37a35f9ab76de709Join with the run's trace
policy_version, context_version2026.2, ctx-2026.11Which rules, which configuration
retrievalquery, method, rerank, k, pool, filters before similarityHow the passages were found
budgettotal, shares, limits, usedHow the room was divided
included23 items: id, evidence id, kind, share, tokens, reasonWhat the model saw
omitted20 items: id, evidence id, kind, tokens, omission reason, detailWhat was left out, and why

On this site one run has one trace. Each attempt and each model call is a span inside it, so the manifest is joined to the trace by trace_id and to its model call by span_id. Case UW-006 shows why the attempt matters: the provider times out, the agent retries, and the run run-e3b4 has two manifests, one per attempt, with the same trace_id and two different span_id values. The second manifest holds both tool results, two timeouts, and the stand-in's request_info follows from them, which is how a reviewer can see that an error is context too.

The standard for the trace side is the OpenTelemetry semantic conventions for generative AI. Read for this page, they have moved from the opentelemetry.io specification page to their own repository, and their status is Development. For a model call they define a gen_ai.client.inference span, named {gen_ai.operation.name} {gen_ai.request.model}. gen_ai.operation.name and gen_ai.provider.name are required; gen_ai.request.model, gen_ai.usage.input_tokens, gen_ai.usage.output_tokens, gen_ai.usage.cache_read.input_tokens, gen_ai.response.id and gen_ai.response.finish_reasons are among the conditionally required or recommended attributes. The content of the window (gen_ai.system_instructions, gen_ai.input.messages, gen_ai.tool.definitions) is opt-in, with a warning that it is likely to hold sensitive data, and capturing it is gated by an explicit user opt-in; the documentation names the environment variable OTEL_INSTRUMENTATION_GENAI_CAPTURE_CONTENT for that switch. There is also a recommended boolean, gen_ai.conversation.compacted, set to true only when an instrumentation can tell that compaction was applied.

What the conventions read here do not have is a place for the omitted items and their reasons. The trace can say which tokens were sent and, if you opt in, the full text. It cannot say that EV-GUIDE-0201 was kept out as superseded and would have ranked third. So the design of this site is two records joined by ids: the span from the conventions, and the manifest stored next to it with the same trace_id and span_id. A sketch of the manifest's summary as one log line (documented shape of the attribute names above; the context.* fields are this site's design; not run here):

{
  "trace_id": "0c17c0046d4371b42858072814d7b4ca",
  "span_id": "37a35f9ab76de709",
  "gen_ai.operation.name": "chat",
  "context.submission_id": "SUB-1004",
  "context.run_id": "run-7f3a",
  "context.attempt_number": 1,
  "context.step": "recommend",
  "context.policy_version": "2026.2",
  "context.context_version": "ctx-2026.11",
  "context.included": 23,
  "context.omitted": {"not_requested": 9, "superseded": 6, "duplicate": 4, "below_rank_cutoff": 1},
  "context.tokens_used": 2288,
  "context.manifest_ref": "manifests/run-7f3a/1/recommend.json"
}

The counts go on the log line for dashboards and alerts. The full manifest, with item ids and details, goes to storage under a key built from the run, the attempt and the step. Neither holds the text of the window. That is a deliberate choice: the manifest is a few kilobytes of ids, while the window holds submission data and confidential guidance, and its retention is a policy decision.

What this is not. This page says what the context record must contain and how it joins a trace. Traces, metrics, log pipelines, sampling, retention and audit events in depth belong to Observability Engineering. The runtime view of the same signals is on Harness Engineering, observability, and turning production traces into evaluation cases is on Eval Engineering, production.

Incident analysis from a manifest#

Incident analysis here means reconstructing, from records, why one model call produced the answer it did, and naming the stage that let the cause in. Back to the reviewer's question about UW-201 in October.

The script below, written for this page, does what the reviewer would do. It reads the stored manifest of that run (written by run.py --context-version ctx-2026.10), rebuilds the window from the same inputs and the same context version, reads the roof limits the model could see, and compares the item fates with the current version. A successful run names the cause without guessing.

1. The stored manifest
   SUB-1201 run-d383 attempt 1 step recommend
   trace aa987d7d6d630edf92e28f3d5613ece8 span dd8e701b95cac23e
   policy 2026.2 context ctx-2026.10 mode plain
   retrieval: lexical, filters before similarity False
   policy passages included:
     EV-NOTE-1701   2026.1  current    tenant-b  underwriting
     EV-GUIDE-0201  2025.4  superseded tenant-a  underwriting
     EV-MEMO-1601   2026.2  current    tenant-a  reinsurance
     EV-GUIDE-0101  2026.2  current    tenant-a  underwriting
     EV-BULL-1101   2026.2  current    tenant-a  underwriting
2. Replay: rebuild the window from the same inputs and context version
   manifest identical to the stored one: True
   window sha256 5df86ed3cbda3ce6
3. What the model could read: roof limits in window order
     EV-NOTE-1701   R-ROOF-01 max_age 35 zone C
     EV-GUIDE-0201  R-ROOF-01 max_age 30 zone C
     EV-GUIDE-0101  R-ROOF-01 max_age 25 zone C
   stand-in recommends approve []
4. The same submission under the current context version
     EV-GUIDE-0102  ctx-2026.10 absent             ctx-2026.11 in
     EV-GUIDE-0201  ctx-2026.10 in                 ctx-2026.11 superseded
     EV-MEMO-1601   ctx-2026.10 in                 ctx-2026.11 unauthorized
     EV-NOTE-1701   ctx-2026.10 in                 ctx-2026.11 other_tenant
   stand-in recommends decline ['R-ROOF-01']

Read it as the reviewer would.

  1. The record places the call. Submission, run, attempt, step, trace and span; policy 2026.2, so the right rules were in force; context version ctx-2026.10, lexical retrieval with no filters before similarity.
  2. The window can be rebuilt. The rebuilt manifest is identical to the stored one, so the reviewer is looking at the window the agent saw, not at a guess.
  3. The window held three roof limits, and the wrong one came first. Another tenant's note (35 years) was the first roof rule in the window, then the superseded 2025.4 rule (30 years), then the current rule (25 years). In plain text, with no labels, the stand-in read the first limit. A roof of 27 years is under 35. It recommended approve, with no citation.
  4. The cause is a stage, not the model. Three items that the current version omits were in the window: other_tenant, superseded and unauthorized. The fix was already shipped as ctx-2026.11, and the same submission under it gives decline citing R-ROOF-01.

The answer to the reviewer fits in one sentence: the October window let in another tenant's roof rule and the superseded rule because the first context version filtered neither before retrieval, and the agent applied the first limit it read. Every part of that sentence comes from a record.

Compare the two versions for any case yourself:

Casectx-2026.10ctx-2026.11What differs (examples)
UW-201approve (expected decline)decline, R-ROOF-01EV-NOTE-1701 in → other_tenant; EV-MEMO-1601 in → unauthorized; EV-GUIDE-0201 in → superseded
UW-202approvedecline, R-ROOF-01EV-GUIDE-0101 over_budget → in; the provider report trimmed to 419 of 4,080 tokens
UW-005approvedecline, R-ROOF-01P-NOTE-17 pasted as plain text → delimited and labelled external_untrusted
UW-004escalateescalate, R-ROOF-03many fates differ; the recommendation does not

The agent here is a scripted stand-in, so these results show how the mechanism works, not how good any model is.

Production pitfall. An incident review that starts from today's configuration answers the wrong question. Rerunning UW-201 today gives decline and suggests there was never a problem. The manifest's context_version is what lets you rebuild October instead of November.

Replay#

Replay means rebuilding the window of a past model call from its recorded inputs and versions, and, where useful, running the model on it again. Step 2 above is a replay of the window: same inputs, same context version, identical manifest. It works because the example's assembler is deterministic, a property its unit tests check: the same inputs and the same context version give the same manifest.

Replay has three levels, and each needs more to be stored.

LevelWhat you getWhat must be kept
Rebuild the manifestWhich items were in and out, and whyThe inputs by version: submission, policy version, index snapshot, decisions as of that day, the context version
Rebuild the window textExactly what the model readAlso the content of every item at that version (or the window itself)
Rerun the modelWhether the same window still gives the same answerAlso the model version and its settings; and even then a real model may answer differently

The third level is the weakest. In the example, UW-010 sends the same window 20 times and gets escalate 15 times. A replay that gives a different answer does not prove the original answer wrong, and one that gives the same answer does not prove it was certain. Replay the window to explain; rerun the model only to compare distributions, which is evaluation method and belongs to Eval Engineering.

The hard requirement hidden in the first row is the index snapshot. If the index is refreshed in place and the old chunks are gone, the October window cannot be rebuilt. Either keep chunk versions (never overwrite a chunk; add a new version and mark the old one), or store the window hash and the item texts for the calls you may need to explain.

Versioning the parts: what a context version must pinContent hashes, index snapshots, and rollout of a new version

Name by content, not by date alone. ctx-2026.11 is a readable name. Behind it, store a hash of the full configuration (retriever settings, embedding model id and version, reranker id, k, thresholds, budget and shares, label format, filter definitions). Two names with the same hash are the same version; one name with two hashes is a bug in your release process.

Pin what the configuration points to. An embedding model referenced by a moving alias is not pinned. A tokenizer that a provider updates changes every budget computation. The facts behind each pin come from the provider's documentation, dated, as on Frameworks.

Snapshot the index by generation. Give every index refresh a generation id and record it in the manifest next to the context version. A chunk row carries the generation that wrote it and the generation that superseded it. Then "the index as of run X" is a query, not an archaeology project.

Roll out a context version like a schema migration. Build both versions' windows for a sample of real traffic (shadow assembly: the new version builds a manifest, the old version's window is sent). Compare manifests: omission-reason distributions, changed items per call, tokens. Then compare answers on the calls whose windows differ. Only then switch. The diff lab above is the smallest version of that comparison.

Embedding model changes are migrations. Query and stored vectors must come from the same model and version. Re-embed the whole corpus into a new index generation, run retrieval measures on both, and switch the context version and the index generation together.

For a Principal engineer: retention, privacy, cost and ownershipWhat to keep, for how long, and who answers the reviewer

Ids, not text, by default. A manifest of ids, sizes and reasons is small and holds no submission data. Full windows hold personal and confidential data, and the OpenTelemetry conventions read for this page put content attributes behind an explicit opt-in for the same reason. Keep manifests for every call for as long as decisions must be explainable; keep window text only where policy allows, encrypted, for a bounded time, and rebuild the rest from versioned inputs.

Tenant boundaries apply to telemetry. A manifest of tenant A lists the ids of items it omitted, including the ids of other tenants' chunks that would have ranked high. Decide whether an id from another tenant may appear in tenant A's record at all; the example records it, which is useful for the platform team and a disclosure for a tenant-facing view. Redact before exposing manifests outside the platform.

Alert on the distribution, not the event. Single omissions are normal. Alerts belong on changes: the share of superseded drops to zero (the freshness filter broke), over_budget doubles (a tool output grew), the mean number of changed items per call jumps on a day with no release (an index refresh or a provider change).

One owner for the answer. Policy versions belong to the policy team, the index to a data team, the context version to the agent team, the model to a platform team. An incident crosses all four. The manifest is the shared record that lets them answer the reviewer together; make its schema a contract with an owner and a version of its own.

A checklist for context in production#

CheckWhy
Every assembly setting lives in a named, hashed context versionA window can be tied to the configuration that built it
Every manifest carries policy_version, context_version and the index generationA wrong answer can come from rules, configuration or data
Every model call writes a manifest, joined to its span by trace_id and span_idThe trace says what was sent; the manifest says what was left out
Omission-reason counts are a dashboard, per context versionDrift shows in the manifests before it shows in the answers
Index refresh marks superseded and withdrawn items in the same jobA stale or withdrawn chunk must not stay current
Chunks are versioned, not overwrittenA past window can be rebuilt
Assembly is deterministic, and replay of a stored manifest is testedIncident analysis starts from October, not from today
Window text is captured only by an explicit, bounded choiceManifests explain; windows disclose

In one breath#

A window is assembled from parts that different teams change on different days: instructions, policy, index, past decisions, retriever, embedding model, budget, model. Give the assembly configuration a context version, record it with the policy version and the index generation in a manifest for every model call, and join that manifest to the trace by trace_id and span_id, because the trace cannot say what was left out. Watch the omission reasons for drift, keep chunk versions so the past can be rebuilt, and answer an incident by replaying the window it actually had, not by rerunning today's.

What comes next#

Every piece of the example has now been taken apart on its own page: the window, the signals, the assembler, retrieval, RAG, the budget, compaction, the coding agent, memory, dynamic context, isolation, security, measurement, frameworks and production. The last page, the Insurance lab, puts them back together: one submission, SUB-1004, from 63 fields, 20 documents and 3,000 past decisions to one window, with its manifest, its budget, its loss test, its measurements, the full catalog of sixteen cases and the code to run it all.