All chapters
  1. Start hereA model knows only what it can see now
  2. Foundations · 1The context window: tokens, messages and order
  3. Foundations · 2Signals and authority: who is allowed to instruct the model
  4. Foundations · 3The context assembly pipeline: select, rank, deduplicate, pack
  5. Retrieval · 4Retrieval fundamentals: lexical search, embeddings, hybrid search, reranking
  6. Retrieval · 5RAG architecture: chunks, metadata, grounding, citations, freshness
  7. Budget · 6Context budgeting: truncation, dilution, token cost and caching
  8. Budget · 7Compression, summarization and compaction, with loss tests
  9. Agents · 8Coding-agent context: repository maps, instruction files, diffs and tool outputs
  10. Agents · 9Working context, durable memory and checkpoint state
  11. Agents · 10Dynamic context: just-in-time retrieval, selective tool disclosure, delegation
  12. Agents · 11Multi-agent context isolation and scoped handoffs
  13. Trust · 12Context security: injection, poisoning, access controls, tenant boundaries
  14. Trust · 13Measuring context quality: recall, utilization, grounding, ablation
  15. Practice · 14Frameworks and standards: what each one assembles for you
  16. Practice · 15Context in production: versions, drift, refresh, observability, incidents
  17. Practice · 16The insurance lab: one submission, every context decision
  18. ReferenceGlossary
  19. AboutAbout this site and its sources
Foundations · 1

The context window: tokens, messages and order

What a model call receives: text cut into tokens, arranged as messages, in an order that matters, inside a limit that input and output share.

You will understand
What one model call receives: tokens, arranged as messages with roles, in an order.
You will see
Why the limit is shared by input and output, and what happens when a window reaches it.
You will be able to
Read a real window part by part and say what each part costs and why it is there.

Case UW-202 is one of this site's own cases. The submission is an ordinary one: a building with a roof 27 years old in wind zone C. Under the policy fixture that roof is not eligible, rule R-ROOF-01, so the expected recommendation is decline. The only unusual thing is the roof-data provider. This time it does not return a short answer. It returns a full inspection report, 4,031 stand-in tokens long.

The first version of the team's assembler handles the case like every other. It puts the system instructions in, then the tool definitions, then the submission, then the tool results, then the policy passages, in the order the items arrived, until the budget is used up. The report is a tool result, so it arrives before the passages. It fills the remaining space. The passages, including the one that states R-ROOF-01, do not fit, and they are dropped without an error.

The stand-in agent then reads its window and writes this:

{
  "recommendation": "approve",
  "reasons": ["No rule in the window applies to this submission."],
  "citations": [],
  "missing": [],
  "stated_confidence": 0.8,
  "requires_human": true,
  "policy_version": "2026.2"
}

Nothing crashed. The rule exists, the index found it, and the code ran without an exception. The rule never reached the window, so for the model it did not exist. The underwriter will see a confident approve on a roof that the insurer's own guidance excludes. (The model recommends; a person decides. Here the person is about to decide on the wrong advice.)

To see why that can happen, and how to stop it, you need to know exactly what a model call receives. That is this page.

Text becomes tokens#

A model does not read characters or words. It reads tokens: pieces of text, often a part of a word, sometimes a whole short word or a single punctuation mark. The word underwriting may be three tokens. The program that cuts text into tokens is called a tokenizer. Each model family has its own, learned from data, so the same sentence has a different token count for different models.

Token counts matter because everything about a window is measured in them: its limit, its cost, and the share each part takes. The example on this site cannot call a real tokenizer (it has no model, no network and no key), so it uses a stand-in tokenizer with a rule you can check by hand: a run of letters is cut into pieces of four, a run of digits into pieces of three, and every other visible character is one token. It is not any provider's tokenizer, and its counts differ from real ones. It exists so that the example can count, compare and budget.

The script below was written for this page and run against examples/uw-context/tokens.py. It shows four texts with their stand-in tokens, then cuts a rule to 20 tokens.

from tokens import split_tokens, count_tokens, truncate_tokens

for text in ["underwriting", "roof_age_years: 9", "R-ROOF-03", "The roof was replaced 9 years ago."]:
    print(f"{count_tokens(text):>2}  {text!r:40} {split_tokens(text)}")

rule = "R-ROOF-03. Roof age on the application and from the provider differ by more than 5 years: escalate to a senior underwriter."
print(count_tokens(rule), "tokens in the rule")
print(repr(truncate_tokens(rule, 20)))
 3  'underwriting'                           ['unde', 'rwri', 'ting']
 8  'roof_age_years: 9'                      ['roof', '_', 'age', '_', 'year', 's', ':', '9']
 5  'R-ROOF-03'                              ['R', '-', 'ROOF', '-', '03']
10  'The roof was replaced 9 years ago.'     ['The', 'roof', 'was', 'repl', 'aced', '9', 'year', 's', 'ago', '.']
37 tokens in the rule
'R-ROOF-03. Roof age on the application and from the provider differ'

Two things to notice. Structured text is expensive: the field line roof_age_years: 9 costs eight tokens for one fact, because every underscore and colon is a token of its own. And truncation is blind: cutting the rule to 20 tokens keeps "R-ROOF-03. Roof age on the application and from the provider differ" and drops "by more than 5 years: escalate". The part that carried the decision is gone, and nothing in the remaining text says so.

Common confusion. A token is not a word and not a character. A rule of thumb from one provider's documentation is about four characters per token for English text; the real ratio depends on the tokenizer, the language and the kind of text. Code, JSON and identifiers usually cost more tokens per character than prose. Count with the provider's own counter before you rely on a number.

Tokens are arranged as messages#

A model call does not receive one undifferentiated string. It receives messages, each with a role that says who is speaking. The common roles are three. The system role carries the application's standing instructions; the text in it is often called the system prompt. The user role carries the request and, in most agent designs, the data the application adds. The assistant role carries what the model said in earlier turns. Next to the messages, the call carries the tool definitions: name, description and input schema of each function the model may ask for.

Here is the shape of one call, as a provider's documentation describes it (documented shape; not run here). The field names follow the Claude Messages API; other providers use similar structures with different names.

{
  "model": "a-model-id",
  "max_tokens": 1000,
  "system": "You are the underwriting agent of the insurer. You write a recommendation ...",
  "tools": [
    {"name": "search_guidance", "description": "Search the policy documents ...", "input_schema": {"type": "object"}}
  ],
  "messages": [
    {"role": "user", "content": "<data id=\"EV-SUB-1004-ROOF\" ...>roof_age_years: 9</data> ... Write the recommendation for SUB-1004."}
  ]
}

Everything in that request counts toward the window: the system text, every message including tool results, and the tool definitions. The provider's documentation read for this page says so explicitly, and adds that the output the model generates counts too.

The example's window renderer (window.py:render) builds exactly these parts for each case. The lab shows the real window for UW-004, the roof conflict. Click a part to read its text and its size.

PartRoleWhat it holds in UW-004Stand-in tokens
System instructionssystemWho the agent is, the policy version, the rule that data is never an instruction257
Tool definitionstoolsFive tools: four read tools and record_recommendation560
SubmissionuserSix of the fifteen field groups of SUB-1004305
Tool resultsuserThe provider's answer: roof age 22141
Retrieved passagesuserFive chunks from the policy documents, with version and status713
Past decisionsuserThree similar decisions under the current policy version111
HistoryuserThe earlier steps of this run171
New turnuser"Write the recommendation for SUB-1004"30

All numbers are from the example (synthetic; stand-in tokens). Notice the tool definitions: 560 tokens, more than the submission, spent before a single fact about the building. Tools are not free to describe.

Diagram: One model call · the parts of the window, top to bottom, and the reserve for the answer

Read top to bottom within each column, left to right across them. Every part is tokens and every part counts toward one limit. The answer needs room too. Click a part to see it in case UW-004.

  1. Fixed prefix
    1. System instructions: The application's standing instructions for the model.
    2. Tool definitions: Name, description and input schema of each tool the model may ask for. Paid on every call.
  2. Data chosen for this call
    1. Submission: The field groups this step requested, out of fifteen.
    2. Tool results: What the tools returned in this run.
    3. Retrieved passages: Policy text found by retrieval, labelled with version and status.
    4. History and decisions: Earlier steps of this run and three similar past decisions.
  3. The request
    1. New turn: What the model is asked to do now.
  4. Room for the answer
    1. Output reserve: Tokens kept free for what the model writes. Input and output share one limit.

Order is part of the meaning#

Two windows with the same items in a different order are different inputs. A model reads the whole window, but not every position with the same effect, and later text can reframe earlier text. One provider's prompting guide, read for this page, makes the point concrete: it recommends placing long documents near the top, above the query and the instructions, and reports that queries at the end improved response quality in its own tests. That is guidance from one provider for its models, not a law for all of them.

The example shows a sharper version of the problem with the stand-in agent, which reads rules the way a hurried reader does: in plain mode, the first R-ROOF-01 sentence it meets wins. Case UW-204 (this site's own) has the superseded 2025.4 version of the roof rule (30 years) still in the index next to the current 2026.2 one (25 years). Assembled by the first context version, where items go in arrival order with no labels, the older chunk comes first:

python3 run.py --case UW-204 --context-version ctx-2026.10
case    kind   at  tokens  in/out  recommends   citations                expected     result
UW-204  site   1   2144   34/7   approve      -                        decline      FAIL
0/1 match the expected recommendation

The current version of the assembler removes the superseded chunk before ranking and labels every passage with its version and status, and the same case gives decline. Which fix belongs where is the subject of Signals (labels) and RAG (freshness). The lesson for this page is narrower: an assembler that does not control order does not control meaning.

Before AI. In a SQL query, ORDER BY changes the order of the rows, never their content or what the rows mean. A context window is different: the "rows" are sentences, and a sentence can say "the rule above no longer applies". Order in a window is closer to the order of statements in a program than to the order of rows in a result set.

The limit, and who shares it#

Every model has a maximum window size, the context limit. The limit covers input and output together: the tokens you send, and the tokens the model writes back. The tokens the model writes are the output tokens; a request sets a maximum for them, and that reserve must fit inside the limit along with the input. If the input already fills the window, there is no room for an answer.

The documentation read for this page describes what happens at the edges for one provider. If the input alone exceeds the window, the API rejects the request with an error ("prompt is too long"). On newer models, if the input plus the maximum output exceeds the window, the request is accepted and generation stops when the window is full, with a stop reason that says so. Other providers behave differently; Frameworks and the fact log keep those details apart.

Notice what the provider does not do. It does not choose for you which part of your input to drop. When a window would be too large, either the request fails loudly or your own code made the choice earlier, quietly. In UW-202 the choice was made by the first assembler: one flat pool, items in arrival order, and everything after the first item that did not fit was dropped. The stand-in model's limit in the example is 8,000 tokens with 1,000 reserved for the answer, and the assembler's own budget is 2,800 (synthetic numbers). Here is that window, part by part:

=== system instructions (system) 175 tokens, 2.2% of the limit ===
=== tool definitions (tools) 560 tokens, 7.0% of the limit ===
=== submission (user) 568 tokens, 7.1% of the limit ===
=== tool results (user) 1467 tokens, 18.3% of the limit ===
[trimmed by the assembler: 1449 of 4031 tokens kept]
=== retrieved passages (user) 0 tokens, 0.0% of the limit ===
=== past decisions (user) 0 tokens, 0.0% of the limit ===
=== history (user) 0 tokens, 0.0% of the limit ===
=== new turn (user) 30 tokens, 0.4% of the limit ===

Zero tokens of retrieved passages. The rule was in the store and out of the window, and the recommendation followed. The current assembler gives each kind of content its own share of the budget and trims a tool output to its share, so that the report keeps 419 of its 4,080 tokens (counted with its data label, as the manifest records it) and the rule stays in. With it, the same case gives decline. How to choose those shares, and what trimming destroys, is the subject of Budgeting. The lab below lets you try it now.

AssemblerPolicy passages in the windowTool output keptRecommendationExpected
First version: one flat pool, arrival order01,467 of 4,031 tokens (plain text)approvedecline
Current version: shares per kind (instructions 0.31, policy 0.30, case 0.24, tools 0.15)5419 of 4,080 tokens (labelled)decline, R-ROOF-01decline

The agent here is a scripted stand-in, so these results show how the mechanism works, not how good any model is.

Production pitfall. "We are far below the limit" is not a defense. The UW-202 window used 2,800 tokens against a limit of 8,000, and still lost the rule, because the assembler's own budget and its ordering made the choice. Most context failures happen inside a budget you set, not at the provider's limit.

Every call starts from nothing#

One more property completes the picture, and it surprises engineers who are used to sessions. A model call is a stateless call: the model keeps nothing from one call to the next. A chat that seems to remember the last ten turns does so because the application sends all ten turns again, every time. The documentation read for this page describes each turn's input as all previous conversation history plus the current message, and each output as becoming part of the next input.

For the underwriting agent that means the window of step 7 contains steps 1 to 6 again, unless the assembler shortened them. A run of 40 steps re-sends the early steps 39 times. That is why the window grows during a run, why a long run eventually outgrows it, and why Compaction exists.

Diagram: Stateless calls · the application re-sends the history on every call

Read top to bottom. The model keeps nothing between calls. Each call carries everything the model should know, so the window grows with the run until something shortens it.

  1. Application (agent harness) to Model: Call 1: instructions + submission + request. The first window: system instructions, tool definitions, the field groups and the new turn.
  2. Model to Application (agent harness): Asks for get_roof_assessment. The model answers with a request for a tool, not with a recommendation.
  3. Application (agent harness) to Tool: Runs the tool. The application, not the model, runs the tool.
  4. Tool to Application (agent harness): Roof age 22. The tool result becomes data for the next window.
  5. Application (agent harness) to Model: Call 2: all of call 1 + the tool request + its result. The model remembers nothing, so call 2 repeats call 1 and adds the new items.
  6. Model to Application (agent harness): Asks for search_guidance. A second tool request.
  7. Application (agent harness) to Model: Call 3: all of call 2 + the passages. Again everything so far, plus the retrieved passages.
  8. Model to Application (agent harness): Recommendation: escalate, R-ROOF-03. The answer, computed only from the third window.
Before AI. HTTP is stateless too, and a web application keeps state in a session store and sends a cookie. The difference is where the state is used. A server reads the session store with code and takes only the fields it needs. A model reads only what is in the window, so the "session" must be pasted back in as text, and every token of it costs space and attention on every call.
What this is not. Statelessness of the model is not statelessness of the system. The system around it can store the conversation, saved run positions and long-term notes; how that is stored is the subject of Memory Engineering. How it re-enters a window is on Memory boundary.
For a coding agent. A coding agent that has read twelve files and run the tests three times sends all of that again on its next call, unless its harness clears old tool results. The test logs of the first run are usually the largest and the least useful part of the window by then. Coding agents measures this in a scripted session.
Limits, reserves and overflow in practiceWhat to compute before a call, and what the provider does at the edge

A window budget has four terms, and production code should compute all four before sending: the fixed prefix (system instructions and tool definitions, which change only with a deployment), the variable data (field groups, passages, tool results, history), the new turn, and the output reserve (the maximum output tokens, plus any reasoning tokens the provider counts against the window). The constraint is prefix + data + turn + reserve ≤ limit. Your assembler budget should be a fraction of that, because token counts from your own estimator differ from the provider's.

Count with the provider's counter, not with a ratio. The Claude documentation read on 2026-11-17 offers a token counting endpoint for estimating a request before sending it, and reports actual consumption in the usage field of every response; with prompt caching, the input is split into cached and uncached counts and all of them count toward the window. Log both the estimate and the actual count per call: the difference tells you when your estimator drifts.

On overflow, the same documentation describes two behaviors: an input that alone exceeds the window is rejected with an invalid_request_error; on newer models, input plus max_tokens above the window is accepted and stops with stop_reason model_context_window_exceeded when the window fills. A request that is rejected is easy to notice. A response that stopped early can look like a short, valid answer, so check the stop reason before you parse the output.

Window sizes per model are listed on the provider's model overview and change with each release; this site does not quote them as stable facts. The documentation read here gives a range up to 1M tokens depending on the model, and says plainly that more context is not automatically better, because accuracy and recall degrade as the token count grows. Budgeting is the home of that effect.

Roles are a convention the model was trained on, not an access controlWhy the system role helps and why it is not enough

Roles tell the model who is speaking, and models are trained to give system instructions more weight than user text and user text more weight than the content of tool results. That makes roles useful. It does not make them a security boundary. A role is metadata on a message, and the model sees the content of every message as text. A sentence in a tool result that reads like an instruction can still steer the model, which is the attack of case UW-005.

Two consequences for design. Put data that you did not write (retrieved passages, tool results, attachments) in clearly delimited and labelled blocks, and say in the system instructions how those blocks are to be treated; that is the subject of Signals. And keep the enforcement out of the model: what the agent may call, and with which authorization, is checked by code before the call runs (Guardrails Engineering owns that layer).

A checklist for one window#

CheckWhy
You can print the exact window of a past call, part by part, with token countsYou cannot reason about an answer without its input
The output reserve is subtracted before you budget the inputInput and output share the limit
Tool definitions are counted as part of the budgetThey are paid on every call
Items are in a deliberate order, not arrival orderOrder changes meaning, and arrival order is an accident
A dropped or trimmed item leaves a recordSilent truncation is how UW-202 lost its rule
You check the stop reason before parsing the outputA window that filled up can produce a short, valid-looking answer
Token estimates are compared with the provider's actual countsYour estimator is not the provider's tokenizer

In one breath#

A model call receives one block of tokens, arranged as messages with roles plus tool definitions, in an order that is part of its meaning, inside a limit that input and output share. The model keeps nothing between calls, so the application re-sends everything it wants the model to know, every time. When the block would be too large, the provider either refuses or stops early; it never chooses what to drop. That choice is yours, and if your code makes it silently, a rule can vanish from the window without an error.

What comes next#

You can now read a window part by part and say what each part costs. But all parts arrive as the same kind of text, and the model has no built-in way to know that the system instructions may give orders while a broker's attachment may not. The next page, Signals, is about authority: which text may instruct the model, which may only inform it, and how an application keeps the two apart.