In October the underwriting agent ran with its first context version, ctx-2026.10. One of its runs was case UW-201, one of this site's own cases: submission SUB-1201, a masonry warehouse with a roof of 27 years in wind zone C. Under the policy in force, 2026.2, rule R-ROOF-01 says a roof older than 25 years in zone C is not eligible. The agent recommended approve. The model recommends; a person decides, and the underwriter decided. But in November a reviewer asks the question every production system eventually gets: why did the agent recommend cover for a roof the guidance excludes?
By then a lot has changed. The context version is ctx-2026.11. The index was refreshed. Run the same submission today and the agent recommends decline. That answer is useless to the reviewer. The question is about the window of October, and the window of October no longer exists.
Unless it was recorded, or can be rebuilt. This page is about the window over months: what changes, how to give the configuration a version, how to notice drift, what to log on every call, and how to answer the reviewer from a record instead of a guess.
A window is built from parts that change on different days#
Look at what goes into one window of the underwriting agent and ask who changes each part, and when.
| Part | What changes it | How often (illustrative) |
|---|---|---|
| System instruction and tool definitions | A release of the agent | Weeks |
The policy fixture (UW-GUIDE) | The policy team; 2025.4 became 2026.2 on 2026-07-01 | Months |
| The index of the 20 policy documents | An ingestion job: new bulletins, withdrawn notes, a superseded version | Days |
| The 3,000 past decisions | Every underwriter decision adds one | Hours |
| Retriever settings: method, reranker, k | The team that owns the assembler | Weeks |
| The embedding model and the tokenizer | A provider or a platform team | Months |
| Budget shares and filters | The team that owns the assembler | Weeks |
| The model itself | The provider or a configuration switch | Months |
The application code of the agent can stay the same through all of these changes. The windows it builds do not. This is the production problem of context engineering: the input of every model call is a function of many versioned things, and the people who change them are not the people who own the agent.
Before AI. A relational database has the same shape. The query text in your code never changes, but the statistics the optimizer uses are refreshed by a background job, an index is added by another team, and one morning the same query takes a different plan. Good teams record the plan with the slow query. The comparison holds up to the plan: a database plan changes speed, while a changed window changes the answer, and nobody gets an error.
A context version#
A context version is a name for one complete assembly configuration: every setting that decides which items enter a window and how. Two windows built for the same inputs under the same context version must be the same; if a setting changes, the name changes.
The example has two:
| Setting | ctx-2026.10 (first) | ctx-2026.11 (current) |
|---|---|---|
| Field selection | all 15 groups | the 6 requested groups |
| Authorize, freshness | off | on, filters before similarity |
| Retrieval | lexical (BM25), no reranking | hybrid (RRF) + stand-in reranker |
| Deduplication | off | Jaccard ≥ 0.8 |
| Budget | 2,800 tokens, one flat pool | 2,800 tokens in four shares |
| Data labels | plain | labelled <data ...> blocks |
| Cases matching the expected recommendation | 8 of 16 | 16 of 16 |
In the code a context version is a dictionary (assemble.py:CONTEXT_VERSIONS), and every manifest carries its name in context_version, next to policy_version. That pairing matters: the policy version says which rules are in force, the context version says how the window was chosen. A wrong answer can come from either, and you need both to tell which.
What belongs inside a context version is everything that changes a window without changing the inputs: retriever method and its parameters, the embedding model and its version, the reranker, k, the deduplication threshold, the budget and its shares, the label format, the filters. What stays outside, recorded separately in the manifest, is what changes with the inputs: the policy version, the index snapshot, the submission.
For a coding agent. A coding agent's context version is its instruction files, its tool list and the harness settings that load and clear context. A change to AGENTS.md changes every later window in that repository, exactly as a change of budget shares does here. Treat it as a reviewed change, and when an agent's behavior shifts, check first whether the instruction file changed. Coding agents shows what those files put in the window.
Drift: the window changes while nobody changes the agent#
Context drift is a change in the windows an agent receives that comes from a change in its parts, not from a change in its inputs or its code. Three kinds matter in the example.
The policy version changes. On 2026-07-01 the roof limit fell from 30 to 25 years. Everything derived from the old version became stale at that moment: the 2025.4 chunks (case UW-204) and the past decisions made under 2025.4 (case UW-007, a similar roof approved under the old rule). Of the 3,000 past decisions in the example, 2,347 were made under 2025.4. The freshness stage keeps them out with the omission reason superseded; without it, the most similar past case for UW-007 would argue for approval.
The index is refreshed. Index refresh is the job that brings the retrieval store up to date with its sources: new documents chunked and embedded, changed ones replaced, withdrawn ones removed, superseded ones marked. Each refresh can change what ranks in the top k for every query. A refresh that adds documents but never removes or re-marks them is the common failure: a withdrawn bulletin keeps answering, or a superseded rule stays current because its status was not updated in the same job that published the new one (RAG lists the ingestion rules).
The retriever or the embedding model changes. A new embedding model ranks the same chunks differently. The example cannot swap in a real model, but it can swap the method. The script below, written for this page, runs all sixteen cases under three settings and counts the omission reasons in their manifests; the third setting is ctx-2026.11 with the retriever switched to vector search only, as a stand-in for an embedding change. A successful run shows drift that the pass rate cannot see.
omitted items per reason, all 16 cases (final attempt)
reason ctx-2026.10 ctx-2026.11 ctx-2026.11 + vector only
not_requested 0 144 144
unauthorized 0 8 6
other_tenant 0 11 11
superseded 0 80 78
duplicate 0 55 60
below_rank_cutoff 105 20 15
over_budget 11 6 6
ctx-2026.10 mean tokens 2111.6 match 8/16
ctx-2026.11 mean tokens 2254.5 match 16/16
ctx-2026.11 + vector only mean tokens 2246.3 match 16/16
ctx-2026.11 against the same version with the retriever swapped (vector only)
UW-001: +1 -1 items decline -> decline
UW-002: +2 -2 items approve -> approve
UW-003: +1 -1 items request_info -> request_info
UW-005: +1 -1 items decline -> decline
UW-007: +1 -1 items decline -> decline
UW-009: +1 -1 items approve -> approve
UW-201: +1 -1 items decline -> decline
UW-202: +1 -1 items decline -> decline
UW-204: +1 -1 items decline -> decline
UW-205: +1 -1 items decline -> decline
UW-206: +1 -1 items decline -> decline
windows changed: 11/16; recommendations changed: 0/16
Two lessons. First, the retriever change altered the window of 11 of 16 cases and no recommendation. Every answer-level check stays green. The next policy change or the next unusual submission may be the one where the swapped passage was the needed one, and the Measuring page showed how quiet that failure is: a missing rule produces a confident approve, not an error. Drift is visible in the manifests before it is visible in the answers.
Second, the distribution of omission reasons is a fingerprint of a context version. ctx-2026.10 omits almost everything as below_rank_cutoff and over_budget; ctx-2026.11 omits mostly by selection, freshness and deduplication. A dashboard of omission reasons per day, per context version, shows a broken filter (the superseded count drops to zero), a growing tool output (over_budget rises) or a flood of copies (duplicate rises) on the day it happens.
Diagram: From changing parts to an explained answer · versions, record, replay (UW-201)
Read left to right. Parts change on different days; a context version names the configuration; every model call writes a manifest joined to its span; an incident is answered by rebuilding the window from the record, not by rerunning today's configuration.
- 1 · Parts that change
- Policy version: The rules in force; everything made under the old version becomes stale on the effective date.
- Index refresh: New, changed, withdrawn and superseded chunks reach the store.
- Retriever and embedding: A new method or embedding model reorders the same chunks.
- 2 · Name the configuration
- Context version: One name for every setting that decides which items enter a window.
- 3 · Record every call
- Manifest: Ids, versions, included items and omitted items with reasons, for one model call.
- Span: The model call in the run's trace, with the attributes of the OpenTelemetry GenAI conventions.
- 4 · Explain later
- Replay: Rebuild the window of a past call from its recorded inputs and versions.
- Incident analysis: Read the rebuilt window, find what misled the answer, name the stage that let it in.
What to log per call#
Context observability is the ability to answer, for any past model call, what its window contained, what was left out, and why, from records kept at the time. It needs one record per model call, and the manifest of this site is designed to be that record.
The manifest of UW-004 carries the identifiers that place it, the versions that explain it, and the two lists:
| Field | Value in UW-004 | Why it is logged |
|---|---|---|
submission_id | SUB-1004 | The business record the underwriter sees |
run_id, attempt_number, step | run-7f3a, 1, recommend | Which model call of which execution |
trace_id, span_id | 0c17c0046d4371b42858072814d7b4ca, 37a35f9ab76de709 | Join with the run's trace |
policy_version, context_version | 2026.2, ctx-2026.11 | Which rules, which configuration |
retrieval | query, method, rerank, k, pool, filters before similarity | How the passages were found |
budget | total, shares, limits, used | How the room was divided |
included | 23 items: id, evidence id, kind, share, tokens, reason | What the model saw |
omitted | 20 items: id, evidence id, kind, tokens, omission reason, detail | What was left out, and why |
On this site one run has one trace. Each attempt and each model call is a span inside it, so the manifest is joined to the trace by trace_id and to its model call by span_id. Case UW-006 shows why the attempt matters: the provider times out, the agent retries, and the run run-e3b4 has two manifests, one per attempt, with the same trace_id and two different span_id values. The second manifest holds both tool results, two timeouts, and the stand-in's request_info follows from them, which is how a reviewer can see that an error is context too.
The standard for the trace side is the OpenTelemetry semantic conventions for generative AI. Read for this page, they have moved from the opentelemetry.io specification page to their own repository, and their status is Development. For a model call they define a gen_ai.client.inference span, named {gen_ai.operation.name} {gen_ai.request.model}. gen_ai.operation.name and gen_ai.provider.name are required; gen_ai.request.model, gen_ai.usage.input_tokens, gen_ai.usage.output_tokens, gen_ai.usage.cache_read.input_tokens, gen_ai.response.id and gen_ai.response.finish_reasons are among the conditionally required or recommended attributes. The content of the window (gen_ai.system_instructions, gen_ai.input.messages, gen_ai.tool.definitions) is opt-in, with a warning that it is likely to hold sensitive data, and capturing it is gated by an explicit user opt-in; the documentation names the environment variable OTEL_INSTRUMENTATION_GENAI_CAPTURE_CONTENT for that switch. There is also a recommended boolean, gen_ai.conversation.compacted, set to true only when an instrumentation can tell that compaction was applied.
What the conventions read here do not have is a place for the omitted items and their reasons. The trace can say which tokens were sent and, if you opt in, the full text. It cannot say that EV-GUIDE-0201 was kept out as superseded and would have ranked third. So the design of this site is two records joined by ids: the span from the conventions, and the manifest stored next to it with the same trace_id and span_id. A sketch of the manifest's summary as one log line (documented shape of the attribute names above; the context.* fields are this site's design; not run here):
{
"trace_id": "0c17c0046d4371b42858072814d7b4ca",
"span_id": "37a35f9ab76de709",
"gen_ai.operation.name": "chat",
"context.submission_id": "SUB-1004",
"context.run_id": "run-7f3a",
"context.attempt_number": 1,
"context.step": "recommend",
"context.policy_version": "2026.2",
"context.context_version": "ctx-2026.11",
"context.included": 23,
"context.omitted": {"not_requested": 9, "superseded": 6, "duplicate": 4, "below_rank_cutoff": 1},
"context.tokens_used": 2288,
"context.manifest_ref": "manifests/run-7f3a/1/recommend.json"
}
The counts go on the log line for dashboards and alerts. The full manifest, with item ids and details, goes to storage under a key built from the run, the attempt and the step. Neither holds the text of the window. That is a deliberate choice: the manifest is a few kilobytes of ids, while the window holds submission data and confidential guidance, and its retention is a policy decision.
What this is not. This page says what the context record must contain and how it joins a trace. Traces, metrics, log pipelines, sampling, retention and audit events in depth belong to Observability Engineering. The runtime view of the same signals is on Harness Engineering, observability, and turning production traces into evaluation cases is on Eval Engineering, production.
Incident analysis from a manifest#
Incident analysis here means reconstructing, from records, why one model call produced the answer it did, and naming the stage that let the cause in. Back to the reviewer's question about UW-201 in October.
The script below, written for this page, does what the reviewer would do. It reads the stored manifest of that run (written by run.py --context-version ctx-2026.10), rebuilds the window from the same inputs and the same context version, reads the roof limits the model could see, and compares the item fates with the current version. A successful run names the cause without guessing.
1. The stored manifest
SUB-1201 run-d383 attempt 1 step recommend
trace aa987d7d6d630edf92e28f3d5613ece8 span dd8e701b95cac23e
policy 2026.2 context ctx-2026.10 mode plain
retrieval: lexical, filters before similarity False
policy passages included:
EV-NOTE-1701 2026.1 current tenant-b underwriting
EV-GUIDE-0201 2025.4 superseded tenant-a underwriting
EV-MEMO-1601 2026.2 current tenant-a reinsurance
EV-GUIDE-0101 2026.2 current tenant-a underwriting
EV-BULL-1101 2026.2 current tenant-a underwriting
2. Replay: rebuild the window from the same inputs and context version
manifest identical to the stored one: True
window sha256 5df86ed3cbda3ce6
3. What the model could read: roof limits in window order
EV-NOTE-1701 R-ROOF-01 max_age 35 zone C
EV-GUIDE-0201 R-ROOF-01 max_age 30 zone C
EV-GUIDE-0101 R-ROOF-01 max_age 25 zone C
stand-in recommends approve []
4. The same submission under the current context version
EV-GUIDE-0102 ctx-2026.10 absent ctx-2026.11 in
EV-GUIDE-0201 ctx-2026.10 in ctx-2026.11 superseded
EV-MEMO-1601 ctx-2026.10 in ctx-2026.11 unauthorized
EV-NOTE-1701 ctx-2026.10 in ctx-2026.11 other_tenant
stand-in recommends decline ['R-ROOF-01']
Read it as the reviewer would.
- The record places the call. Submission, run, attempt, step, trace and span; policy
2026.2, so the right rules were in force; context versionctx-2026.10, lexical retrieval with no filters before similarity. - The window can be rebuilt. The rebuilt manifest is identical to the stored one, so the reviewer is looking at the window the agent saw, not at a guess.
- The window held three roof limits, and the wrong one came first. Another tenant's note (35 years) was the first roof rule in the window, then the superseded
2025.4rule (30 years), then the current rule (25 years). In plain text, with no labels, the stand-in read the first limit. A roof of 27 years is under 35. It recommendedapprove, with no citation. - The cause is a stage, not the model. Three items that the current version omits were in the window:
other_tenant,supersededandunauthorized. The fix was already shipped asctx-2026.11, and the same submission under it givesdeclinecitingR-ROOF-01.
The answer to the reviewer fits in one sentence: the October window let in another tenant's roof rule and the superseded rule because the first context version filtered neither before retrieval, and the agent applied the first limit it read. Every part of that sentence comes from a record.
Compare the two versions for any case yourself:
| Case | ctx-2026.10 | ctx-2026.11 | What differs (examples) |
|---|---|---|---|
UW-201 | approve (expected decline) | decline, R-ROOF-01 | EV-NOTE-1701 in → other_tenant; EV-MEMO-1601 in → unauthorized; EV-GUIDE-0201 in → superseded |
UW-202 | approve | decline, R-ROOF-01 | EV-GUIDE-0101 over_budget → in; the provider report trimmed to 419 of 4,080 tokens |
UW-005 | approve | decline, R-ROOF-01 | P-NOTE-17 pasted as plain text → delimited and labelled external_untrusted |
UW-004 | escalate | escalate, R-ROOF-03 | many fates differ; the recommendation does not |
The agent here is a scripted stand-in, so these results show how the mechanism works, not how good any model is.
Production pitfall. An incident review that starts from today's configuration answers the wrong question. RerunningUW-201today givesdeclineand suggests there was never a problem. The manifest'scontext_versionis what lets you rebuild October instead of November.
Replay#
Replay means rebuilding the window of a past model call from its recorded inputs and versions, and, where useful, running the model on it again. Step 2 above is a replay of the window: same inputs, same context version, identical manifest. It works because the example's assembler is deterministic, a property its unit tests check: the same inputs and the same context version give the same manifest.
Replay has three levels, and each needs more to be stored.
| Level | What you get | What must be kept |
|---|---|---|
| Rebuild the manifest | Which items were in and out, and why | The inputs by version: submission, policy version, index snapshot, decisions as of that day, the context version |
| Rebuild the window text | Exactly what the model read | Also the content of every item at that version (or the window itself) |
| Rerun the model | Whether the same window still gives the same answer | Also the model version and its settings; and even then a real model may answer differently |
The third level is the weakest. In the example, UW-010 sends the same window 20 times and gets escalate 15 times. A replay that gives a different answer does not prove the original answer wrong, and one that gives the same answer does not prove it was certain. Replay the window to explain; rerun the model only to compare distributions, which is evaluation method and belongs to Eval Engineering.
The hard requirement hidden in the first row is the index snapshot. If the index is refreshed in place and the old chunks are gone, the October window cannot be rebuilt. Either keep chunk versions (never overwrite a chunk; add a new version and mark the old one), or store the window hash and the item texts for the calls you may need to explain.
Versioning the parts: what a context version must pinContent hashes, index snapshots, and rollout of a new version
Name by content, not by date alone. ctx-2026.11 is a readable name. Behind it, store a hash of the full configuration (retriever settings, embedding model id and version, reranker id, k, thresholds, budget and shares, label format, filter definitions). Two names with the same hash are the same version; one name with two hashes is a bug in your release process.
Pin what the configuration points to. An embedding model referenced by a moving alias is not pinned. A tokenizer that a provider updates changes every budget computation. The facts behind each pin come from the provider's documentation, dated, as on Frameworks.
Snapshot the index by generation. Give every index refresh a generation id and record it in the manifest next to the context version. A chunk row carries the generation that wrote it and the generation that superseded it. Then "the index as of run X" is a query, not an archaeology project.
Roll out a context version like a schema migration. Build both versions' windows for a sample of real traffic (shadow assembly: the new version builds a manifest, the old version's window is sent). Compare manifests: omission-reason distributions, changed items per call, tokens. Then compare answers on the calls whose windows differ. Only then switch. The diff lab above is the smallest version of that comparison.
Embedding model changes are migrations. Query and stored vectors must come from the same model and version. Re-embed the whole corpus into a new index generation, run retrieval measures on both, and switch the context version and the index generation together.
For a Principal engineer: retention, privacy, cost and ownershipWhat to keep, for how long, and who answers the reviewer
Ids, not text, by default. A manifest of ids, sizes and reasons is small and holds no submission data. Full windows hold personal and confidential data, and the OpenTelemetry conventions read for this page put content attributes behind an explicit opt-in for the same reason. Keep manifests for every call for as long as decisions must be explainable; keep window text only where policy allows, encrypted, for a bounded time, and rebuild the rest from versioned inputs.
Tenant boundaries apply to telemetry. A manifest of tenant A lists the ids of items it omitted, including the ids of other tenants' chunks that would have ranked high. Decide whether an id from another tenant may appear in tenant A's record at all; the example records it, which is useful for the platform team and a disclosure for a tenant-facing view. Redact before exposing manifests outside the platform.
Alert on the distribution, not the event. Single omissions are normal. Alerts belong on changes: the share of superseded drops to zero (the freshness filter broke), over_budget doubles (a tool output grew), the mean number of changed items per call jumps on a day with no release (an index refresh or a provider change).
One owner for the answer. Policy versions belong to the policy team, the index to a data team, the context version to the agent team, the model to a platform team. An incident crosses all four. The manifest is the shared record that lets them answer the reviewer together; make its schema a contract with an owner and a version of its own.
A checklist for context in production#
| Check | Why |
|---|---|
| Every assembly setting lives in a named, hashed context version | A window can be tied to the configuration that built it |
Every manifest carries policy_version, context_version and the index generation | A wrong answer can come from rules, configuration or data |
Every model call writes a manifest, joined to its span by trace_id and span_id | The trace says what was sent; the manifest says what was left out |
| Omission-reason counts are a dashboard, per context version | Drift shows in the manifests before it shows in the answers |
| Index refresh marks superseded and withdrawn items in the same job | A stale or withdrawn chunk must not stay current |
| Chunks are versioned, not overwritten | A past window can be rebuilt |
| Assembly is deterministic, and replay of a stored manifest is tested | Incident analysis starts from October, not from today |
| Window text is captured only by an explicit, bounded choice | Manifests explain; windows disclose |
In one breath#
A window is assembled from parts that different teams change on different days: instructions, policy, index, past decisions, retriever, embedding model, budget, model. Give the assembly configuration a context version, record it with the policy version and the index generation in a manifest for every model call, and join that manifest to the trace by trace_id and span_id, because the trace cannot say what was left out. Watch the omission reasons for drift, keep chunk versions so the past can be rebuilt, and answer an incident by replaying the window it actually had, not by rerunning today's.
What comes next#
Every piece of the example has now been taken apart on its own page: the window, the signals, the assembler, retrieval, RAG, the budget, compaction, the coding agent, memory, dynamic context, isolation, security, measurement, frameworks and production. The last page, the Insurance lab, puts them back together: one submission, SUB-1004, from 63 fields, 20 documents and 3,000 past decisions to one window, with its manifest, its budget, its loss test, its measurements, the full catalog of sixteen cases and the code to run it all.