All chapters
  1. Start hereA model knows only what it can see now
  2. Foundations · 1The context window: tokens, messages and order
  3. Foundations · 2Signals and authority: who is allowed to instruct the model
  4. Foundations · 3The context assembly pipeline: select, rank, deduplicate, pack
  5. Retrieval · 4Retrieval fundamentals: lexical search, embeddings, hybrid search, reranking
  6. Retrieval · 5RAG architecture: chunks, metadata, grounding, citations, freshness
  7. Budget · 6Context budgeting: truncation, dilution, token cost and caching
  8. Budget · 7Compression, summarization and compaction, with loss tests
  9. Agents · 8Coding-agent context: repository maps, instruction files, diffs and tool outputs
  10. Agents · 9Working context, durable memory and checkpoint state
  11. Agents · 10Dynamic context: just-in-time retrieval, selective tool disclosure, delegation
  12. Agents · 11Multi-agent context isolation and scoped handoffs
  13. Trust · 12Context security: injection, poisoning, access controls, tenant boundaries
  14. Trust · 13Measuring context quality: recall, utilization, grounding, ablation
  15. Practice · 14Frameworks and standards: what each one assembles for you
  16. Practice · 15Context in production: versions, drift, refresh, observability, incidents
  17. Practice · 16The insurance lab: one submission, every context decision
  18. ReferenceGlossary
  19. AboutAbout this site and its sources
Budget · 6

Context budgeting: truncation, dilution, token cost and caching

A window has a limit and every token in it competes for attention and costs money. How to give each kind of content a share, what truncation destroys, and what a cache needs to stay valid.

You will understand
Why a window needs a budget with shares, and what truncation destroys when it has none.
You will see
Why a bigger window is not the fix: attention, context rot and the tokens a long run re-sends.
You will be able to
Order a window so its prefix stays cacheable, and name the changes that invalidate it.

On The context window the first assembler lost rule R-ROOF-01 in case UW-202, this site's own case. The roof-data provider returned a 4,031-token inspection report, the report took the space, the policy passages were dropped, and the stand-in agent recommended approve for a roof of 27 years in wind zone C. (The model recommends; a person decides. Here the person would have decided on advice built from the wrong window.)

The team's first proposal for a fix was short. "The stand-in model's limit is 8,000 tokens and we budget 2,800. Raise the budget to 7,000 and everything fits." They tried it, and it worked:

setting                       tool result passages R-ROOF-01  recommends
one flat pool of 7,000               4080        5       yes  decline   window 6385

The rule is back and the recommendation is right. The report is in the window in full, all 4,080 tokens of it. The window is now 6,385 tokens, more than twice the size of a normal case, and most of it is facet measurements nobody asked for.

A senior engineer on the team asked three questions before approving the change. What happens when the next report is 9,000 tokens? Does a model read a rule as carefully when it sits next to four thousand tokens of facet data? And what does it cost to send 6,385 tokens on every call of a run that makes forty calls? This page answers the three questions. The short answers: the next report breaks it again, the model may not, and a lot.

A budget, and the shares inside it#

A context budget is the number of tokens a window, or one part of it, is allowed to use. It is a decision your code makes, not a property of the model. The provider's limit is a ceiling; the budget is the plan underneath it. For example: "this step may use 2,800 tokens, and at most 840 of them for policy passages."

The second half of that sentence is the important one. A single total says how much the window may hold. It says nothing about who gets the room. With one flat pool, the first large item takes whatever it needs, and the items that arrive after it get what is left. In UW-202 that was nothing.

The current assembler (assemble.py:pack) cuts the budget into shares, one per kind of content, called buckets in the code:

BucketWhat it holdsShareLimit at 2,800
instructionsSystem instructions, the new turn, tool definitions0.31868
policyRetrieved passages of the policy documents0.30840
caseField groups of the submission, history, past decisions0.24672
toolsTool results0.15420

Each bucket fills in priority order up to its own limit. Leftover room is not lent to another bucket. That last rule looks wasteful, and it is deliberate: it is what makes a share a guarantee. If the tools bucket could borrow from policy, a large report would borrow until the rule was gone again.

Before AI. A JVM has one heap, and one runaway thread can exhaust it for every other thread. Production systems put bulkheads in front of shared resources instead: a connection pool per downstream service, a thread pool per queue, so that one slow dependency cannot starve the others. Budget shares are bulkheads for a window. The comparison stops at recovery: a starved thread pool raises an error you can see, while a starved share produces a window that is shorter than it should be and an answer that looks normal.

Here is the effect on UW-202, four settings of the same assembler. The script was written for this page and run against examples/uw-context/; it assembles the case with each setting and asks the stand-in agent for its recommendation.

from agent import run_agent
from assemble import assemble
from scenario import load_case

case = load_case("UW-202")
settings = [
    ("shares 0.31/0.30/0.24/0.15", {}),
    ("one flat pool", {"shares": None}),
    ("policy 0.05, tools 0.40", {"shares": {"instructions": 0.31, "policy": 0.05, "case": 0.24, "tools": 0.40}}),
    ("one flat pool of 7,000", {"shares": None, "budget": 7000}),
]
print(f"{'setting':28} {'tool result':>12} {'passages':>8} {'R-ROOF-01':>9}  recommends")
for name, over in settings:
    built = assemble(case, "ctx-2026.11", **over)
    inc = built["manifest"]["included"]
    tool = next(i for i in inc if i["kind"] == "tool_result")
    passages = [i for i in inc if i["kind"] == "policy_passage"]
    has_rule = any(i["item_id"] == "EV-GUIDE-0101" for i in inc)
    rec = run_agent(built["window"])["recommendation"]
    print(f"{name:28} {tool['tokens']:>12} {len(passages):>8} {'yes' if has_rule else 'no':>9}  {rec['recommendation']:<9}"
          f" window {built['manifest']['budget']['used_total']}")
setting                       tool result passages R-ROOF-01  recommends
shares 0.31/0.30/0.24/0.15            419        5       yes  decline   window 2612
one flat pool                        1647        0        no  approve   window 2800
policy 0.05, tools 0.40              1120        1        no  approve   window 2675
one flat pool of 7,000               4080        5       yes  decline   window 6385

Read the rows as four designs. The default shares keep all five passages and give the report its 420-token share; the window is 2,612 tokens and the answer is right. A flat pool of the same size lets the report take 1,647 tokens and drops every passage. Shares that starve the policy bucket fail the same way, more slowly: one passage survives, and it is not the rule. A flat pool of 7,000 is right too, at 2.4 times the window, and only because this report happens to fit. Shares make the outcome independent of how long the provider's answer is; a bigger pool only moves the point where it breaks.

(The "one flat pool" row here keeps the current version's labels and filters, so its report keeps 1,647 tokens. The first context version on The context window is plain text and kept 1,467. Same mechanism, different settings.)

The lab lets you move the shares yourself. Start from the default, then try "starve the policy share", then a flat pool, and watch the row "needed rule readable in the window".

Setting (UW-202, 2,800 tokens)PackedRule R-ROOF-01 in windowStand-in recommends
Shares 0.31 / 0.30 / 0.24 / 0.15 (default)report trimmed to 419 of 4,080; one past decision droppedyesdecline (expected)
One flat poolreport trimmed to 1,647; all 5 passages, history and past decisions droppednoapprove
Policy share 0.05, tools 0.40report trimmed to 1,120; 4 of 5 passages droppednoapprove

Share limits at the default: instructions 868, policy 840, case 672, tools 420. The agent here is a scripted stand-in, so these results show how the mechanism works, not how good any model is. All numbers are synthetic stand-in tokens.

Diagram: Budget shares · one total, four bulkheads, and what happens to an item that does not fit

Read left to right. The total is cut into shares before anything is packed. Each share fills on its own, so the long roof report of UW-202 can only lose its own tail, never the policy passages. Click a box to see the numbers of the example.

  1. The budget
    1. Context budget: The tokens this step may use, chosen by the application below the model's limit.
  2. Shares (bulkheads)
    1. Instructions: System instructions, the new turn and the tool definitions. Mostly fixed per step.
    2. Policy: Retrieved passages. Protected room for the rules the answer must cite.
    3. Case: Field groups, history and past decisions of this submission.
    4. Tools: Tool results. The smallest share; long outputs are trimmed to it.
  3. Packing inside each share
    1. Fits: Items in priority order go in while their share has room.
    2. Trimmed: A trimmable tool output is cut to the room left; a visible mark records kept and total tokens.
    3. Dropped: An item that does not fit and may not be trimmed is omitted with a reason in the manifest.
  4. Result
    1. Window and manifest: The window the model reads, and the record of every fit, trim and drop.

Truncation, and what it destroys#

Truncation means cutting text to a size by keeping one end and dropping the rest. A tokenizer-aware cut keeps the first N tokens; the tail is gone. For example, the rule R-ROOF-03 cut to 20 stand-in tokens keeps "Roof age on the application and from the provider differ" and loses "by more than 5 years: escalate". The context window showed that cut; the question here is where truncation is acceptable at all.

Two kinds of cutting happen in a window, and they deserve different names. Dropping removes whole items: a passage is in or out. Trimming shortens one item and keeps its head. The example's packer does both. A passage that does not fit is dropped and recorded as over_budget. A tool result that is marked trimmable is cut to the room left in its share, and a visible mark says how much was kept. This is the trimmed report in the current UW-202 window (run.py --case UW-202 --show window, shortened here: the line ... stands for facets 2 and 3):

=== tool results (user) 419 tokens, 5.2% of the limit ===
<data id="EV-ROOF-1202-A1" kind="tool_result" source="get_roof_assessment" trust="internal_data">
{"tool": "get_roof_assessment", "submission_id": "SUB-1202", "attempt": 1, "status": "ok", "roof_age_years": 27, "source": "roof-data provider (stand-in)", "method": "aerial image, synthetic"}

Detailed roof report for SUB-1202 (synthetic). Summary: estimated roof age 27 years.
Facet 1: slope 16 degrees, area 1176 sq ft, covering metal, granule loss 36 percent, ponding moderate, patch count 8, image tile T-93487, image date 2026-01-28, confidence 88 percent, vegetation overhang heavy.
...
Facet 4: slope 21 degrees, area 440 sq ft, covering shingle
[trimmed by the assembler: 353 of 4031 tokens kept]
</data>

Trimming worked here for one reason that has nothing to do with the packer: the provider puts the roof age in the first line. The fact the agent needs sits at the head, and the head is what a trim keeps. If the provider had put its summary at the end, the same trim would have kept 353 tokens of facet data and dropped the only number that mattered, and the window would show no sign of it except the trim mark.

So trimming is safe only under a condition you have to check, per source: the facts a step needs are near the start of the output. When that is not true, trimming is the wrong tool. The better tools change the output before it reaches the packer: ask the tool for less (a summary field, a page of results instead of all), extract the fields you need in code and pass those, or keep the full output outside the window and put a short reference in it. A coding agent's test log has the opposite shape to the roof report (the failure is at the end), and Coding agents shows what head-trimming does to it.

Production pitfall. A trim with no mark is worse than a drop. A dropped item appears in the manifest as omitted; a silently trimmed item appears as included, and a reviewer reading "the report was in the window" will assume all of it was. Always add a visible trim mark to the text, and record kept and total tokens in the manifest ("reason": "trimmed to fit: 419 of 4080 tokens").

More text can make answers worse#

Now the second question: does a model read a rule as carefully when it sits next to four thousand tokens of facet data?

A model does not read a window the way a parser reads a file, giving every line the same treatment. It weighs every part of the window against every other part, and that weighing has a limited capacity. Anthropic's engineering article on context engineering (2025) describes this as an attention budget: every token added draws on it. When the window holds a lot of text that is not relevant to the question, the relevant sentences receive a smaller share of the model's attention. This site calls that effect attention dilution: the more irrelevant text in the window, the less weight the relevant text gets. A small example: one rule in a 600-token window is a large part of what the model is looking at; the same rule in a 6,000-token window full of facet measurements is a small part.

The same article, and the Claude documentation on context windows read for this page, give the measured version a name: context rot. As the number of tokens in the window grows, the model's ability to recall and use information in it degrades. The documentation states it plainly: more context is not automatically better, and curating what is in the window matters as much as how much space is available. The article adds two details worth knowing. The decline is gradual, a slope and not a cliff. And its stated cause is structural: in a transformer every token can relate to every other token, so the number of relationships grows with the square of the length, and models see fewer very long sequences during training.

Two terms, then, and they are not the same thing. Attention dilution is the mechanism this site uses to reason about one window: irrelevant tokens compete with relevant ones. Context rot is the observed result across lengths: recall gets worse as windows get longer.

Common confusion. "We are far below the limit, so length is free." Context rot is about length, not about the limit. A window at 30 percent of a large limit can still hold enough irrelevant text to bury the one sentence that decides the answer.

The example cannot show context rot. Its stand-in agent is a script that finds a rule sentence with a pattern, whether the window is 600 tokens or 6,000. That is exactly why the flat pool of 7,000 "worked" above: the stand-in has no attention to dilute. A real model reading that window would be reading 4,080 tokens of facet data to find one number in its first line. Treat the decline in that row as what a perfect reader would do, not as evidence that a large window is safe.

What the example can show is how much of a window an answer used. measure.py computes utilization: the share of data items and data tokens the recommendation rested on.

case    rec          used/data items token share  grounded
UW-202  decline            4/16            0.390 1/1

In the current UW-202 window the stand-in agent's answer rests on 4 of 16 data items, 39 percent of the data tokens. The other 61 percent was carried, paid for, and not used. Low utilization does not prove dilution, and some unused items are there on purpose (a passage that confirms nothing changed is still evidence). It is the number to watch: when a change to the assembler adds tokens and utilization falls, ask what the new tokens are for. Measuring is the home of utilization and of ablation, the method that tells you which items carried an answer.

What tokens cost#

The third question was about cost. Token cost is what a call costs in tokens processed: the input tokens the model reads and the output tokens it writes, each billed by the provider at its own rate. This site states no prices; they change, and they differ per model. The structure is stable enough to reason about without them. For example: a call that reads 2,612 tokens and writes 150 costs 2,612 input tokens and 150 output tokens, whatever the rate.

The structure that matters for agents is the one The context window introduced: a call is stateless, so the window of step 7 contains steps 1 to 6 again. The cost of a run is not the size of its last window; it is the sum of every window it sent. The script below models the long run UW-203 (this site's own case: 40 steps for SUB-1004) as one model call per step, each call re-sending the fixed prefix of 817 tokens plus the history so far. One call per step is a simplification, marked as such; the shape of the result does not depend on it.

"""Budgeting page: input tokens processed over the 40 calls of UW-203 when every call re-sends the history."""
from compact import long_run

PREFIX = 817          # tools + system instructions of the underwriting agent (stand-in tokens, from budget_demo.py)
steps = long_run()
sent, new, total_sent, total_new = 0, 0, 0, 0
history = 0
print("call  history  input sent  new since last call")
for s in steps:
    history += s["tokens"]
    sent = PREFIX + history
    new = s["tokens"] if s["step"] > 1 else sent
    total_sent += sent
    total_new += new
    if s["step"] in (1, 10, 20, 30, 40):
        print(f"{s['step']:>4}  {history:>7}  {sent:>10}  {new:>19}")
print(f"over 40 calls: {total_sent} input tokens sent; {total_new} of them were new; "
      f"{total_sent - total_new} repeated an earlier call's prefix ({(total_sent - total_new) / total_sent:.0%})")
call  history  input sent  new since last call
   1       80         897                  897
  10     1084        1901                   23
  20     2676        3493                   38
  30     3876        4693                   31
  40     4541        5358                   13
over 40 calls: 135865 input tokens sent; 5358 of them were new; 130507 repeated an earlier call's prefix (96%)

The last window is 5,358 tokens. The run sent 135,865. Input volume grows with the square of the run length, because each call repeats all the calls before it. And 96 percent of what was sent had been sent before, word for word, at the start of the previous call. That last number is the opening for the next section. (All numbers are stand-in tokens from a synthetic run.)

The levers on cost follow from the structure, in order of how much they usually save: send fewer calls (do deterministic work in code, as check_eligibility does); send smaller windows (shares, selection, the earlier pages); shorten the history (the subject of Compaction); and stop paying full rate for the repeated part (caching, next).

Prompt caching, and the prefix it depends on#

Prompt caching is a provider feature that lets the provider reuse the work it did on the beginning of an earlier request, when a new request begins with exactly the same tokens. You still send the whole window; the provider recognizes the part it has seen and does not process it again. For example: if call 2 begins with the same 817 tokens of tool definitions and system instructions as call 1, those 817 tokens can be read from the cache.

The part that must match is the cache prefix: the leading section of a request, from its first token up to a point the application marks (or the provider chooses), that must be identical to an earlier request for the cache to be used. Identical means identical. The Claude documentation read for this page says a cache hit needs 100 percent identical prompt segments up to and including the marked block; the OpenAI documentation says the entire rendered prefix must match. One changed character in the prefix, and everything after it is processed again.

Three facts from the Claude documentation, read today, shape how a window should be built for caching (documented behavior; not run here):

FactConsequence for the assembler
The cached prefix is read in a fixed order: tools, then system, then messagesTool definitions are the outermost layer; system instructions next
A change at one level invalidates that level and every later one; changing tool definitions invalidates everythingNever vary tool definitions per call; their order is part of the prefix
Caching does not change the output, and cached tokens still count toward the windowCaching saves processing, not space; budgets stay the same

The example has no provider, so it cannot cache. It can show what the provider would see. The script below cuts each window into blocks in the order a cache reads them (five tool definitions, the system instructions, then each data item and the new turn) and counts the tokens of the leading blocks that two requests share. A block matches whole or not at all, which is how the documentation describes it.

from assemble import assemble
from scenario import load_case
from tokens import count_tokens
from window import render_item


def blocks(case_id, version="ctx-2026.11", system_first_line=""):
    """The request as blocks, in the order a prompt cache reads them: tools, then system, then messages."""
    built = assemble(load_case(case_id), version)
    mode = built["manifest"]["mode"]
    tools = [i["text"] for i in built["items"] if i["kind"] == "tool_definition"]
    system = system_first_line + built["window"]["messages"][0]["content"]
    rest = [render_item(i, mode) for i in built["items"] if i["kind"] not in ("tool_definition", "instruction")]
    return tools + [system] + rest


def reusable(a, b):
    """Tokens of the leading blocks that are identical in both requests. A block matches whole or not at all."""
    n = 0
    for x, y in zip(a, b):
        if x != y:
            break
        n += count_tokens(x)
    return n


base = blocks("UW-004")
print(f"UW-004 request: {len(base)} blocks, {sum(count_tokens(b) for b in base)} stand-in tokens")
others = [
    ("UW-004 again, same context version", blocks("UW-004")),
    ("UW-202, same context version", blocks("UW-202")),
    ("UW-004, timestamp on top of the system text", blocks("UW-004", system_first_line="now: 2026-11-17T09:14:03Z\n")),
    ("UW-004, first context version", blocks("UW-004", "ctx-2026.10")),
    ("UW-004, tool definitions in another order", list(reversed(base[:5])) + base[5:]),
]
for name, other in others:
    print(f"  reusable prefix vs {name:44} {reusable(base, other):>5} tokens")
UW-004 request: 23 blocks, 2288 stand-in tokens
  reusable prefix vs UW-004 again, same context version            2288 tokens
  reusable prefix vs UW-202, same context version                   817 tokens
  reusable prefix vs UW-004, timestamp on top of the system text    560 tokens
  reusable prefix vs UW-004, first context version                  560 tokens
  reusable prefix vs UW-004, tool definitions in another order        0 tokens

Five lines, five lessons. The same window twice is reusable in full: that is a call resent after a network error, or, in its general form, the next call of a run whose history only grows at the end (everything before the new part matches). A different submission shares 817 tokens, the tool definitions and the system instructions: the fixed prefix that every call of the underwriting agent begins with. A timestamp at the top of the system text costs the system block: only the tools survive. A new context version changes the system instructions, so a deployment of the assembler resets the cache too (that is expected; it should happen once per release, not once per call). And a different order of the same tool definitions shares nothing.

The rule for the assembler follows: stable content first, in a stable order, byte for byte; anything that varies per call (the submission, the passages, the time, the run id) after it. The window order of the example already does this, which is why the fixed prefix is reusable across all sixteen cases.

Diagram: The cache prefix · stable layers first, and what invalidates each layer

Read left to right: this is the order in which a prompt cache compares a request with an earlier one. A change in one layer invalidates that layer and everything to its right. Put what never changes on the left.

  1. Layer 1: tools
    1. Tool definitions: Compared first. Changing, adding or reordering a definition invalidates everything after it.
  2. Layer 2: system
    1. System instructions: Stable per deployment. A timestamp or a per-call value here breaks the cache for every call.
  3. Layer 3: messages
    1. Case data: Submission, tool results, passages, past decisions. Different for every case, so it comes after the stable part.
    2. History and new turn: Appending keeps the earlier prefix reusable; rewriting the history (compaction, clearing) does not.
  4. Provider
    1. Prompt cache: Reuses the processing of an identical prefix. Same output, same window space, less processing.
What this is not. Caching does not make a window smaller, and it does not help the model read it. The Claude documentation on context windows says cached prefixes still occupy the window: caching changes what you pay for those tokens, not whether they count. Context rot applies to cached tokens exactly as to fresh ones.

There is one tension to know about before the next page. Shortening a history, whether by clearing old tool results or by summarizing, rewrites the middle of the window, so the prefix after the change no longer matches the one in the cache. The Claude documentation on context editing says so for tool-result clearing, and offers a clear_at_least setting so that each clear removes enough tokens to be worth the new cache write. The OpenAI caching guide lists compaction, summarization and truncation among the things that change the prefix. Compaction saves tokens on every later call and costs one cache rebuild now. Compaction is about doing that well.

For a coding agent. The Claude Code documentation read for this page describes the same layers: tool definitions and core instructions first, then CLAUDE.md and project context, then the conversation, and it lists the actions that invalidate the cache, among them switching models, connecting an MCP server whose tools load up front, and compacting. Editing CLAUDE.md in the middle of a session does not invalidate it, because the file is read at session start; the edit takes effect after /clear, /compact or a restart. Coding agents follows a coding agent's window step by step.
Choosing shares: a method, not a constantHow to set shares from data, and the failure modes of each choice

The example's shares (0.31, 0.30, 0.24, 0.15) were chosen for its sixteen cases; they are illustrative. A method that transfers:

  1. Measure the fixed part first. System instructions and tool definitions are the same on every call of a step. Their share is not a choice; it is their size plus a margin for the next release. In the example they are 817 of 868 tokens in the instructions bucket.
  2. Size each variable bucket from the manifests of real cases. For each kind, take a high percentile of the tokens that the items the answer used actually needed (from utilization and ablation, Measuring), not of what was available.
  3. Give tool outputs the smallest share that keeps their decisive fields, and make trimming safe per tool. For every trimmable source, write down where its decisive fields sit. If they are not at the head, do not trim; extract.
  4. Do not lend unused room between buckets by default. Lending makes the share of a bucket depend on the size of another, which is the flat-pool failure again. If you do lend, lend only from buckets whose items are all included, and record the loan in the manifest.
  5. Test with adversarial sizes. Run every case with the largest tool output you have seen, doubled. A share design passes when the required items of every case stay in.

Failure modes worth naming: a share that is too small for its required item (the system instructions do not fit, and the packer must either break the budget or drop instructions; fail loudly instead); shares that sum to more than 1.0 (the budget is silently exceeded); and shares tuned on easy cases (the long report never appeared in the tuning set). The example's unit tests check that included tokens per share stay within the share's limit; copy that invariant.

Dynamic alternatives exist: a budget that grows with the case's difficulty, or a first pass that asks for less and a second that asks for more. They cost a second call and a harder manifest. Dynamic context covers loading on demand, which is often the better answer to a source that is sometimes huge.

Prompt caching across three providers, as documented todayMechanics that differ, mechanics that agree, and what to log

All rows from each provider's documentation read on 2026-11-17 (documented behavior; not run here; no prices). Details change with model generations; re-read before you rely on one.

QuestionClaudeOpenAIGemini
On by default?No: cache_control at the top level (automatic) or on blocks (explicit), up to 4 breakpointsYes, for supported models; explicit breakpoints on newer modelsImplicit caching on by default for 2.5 and newer
What must match100 percent identical segments up to the marked block, in the order tools, system, messagesThe entire rendered prefix up to a breakpointA similar prefix; put large common content first
Minimum sizeVaries by model, from 512 to 4,096 tokens in the table read; shorter prompts are not cached and no error is returned1,024 tokens on newer models2,048 or 4,096 tokens depending on the model
Lifetime5 minutes by default, refreshed on each use; a 1-hour optionOptions per model generation (for example 30 minutes or 12 hours on newer models)Explicit caches have a time to live; the page read does not state a default
How you see itcache_read_input_tokens, cache_creation_input_tokens, input_tokensusage.input_tokens_details.cached_tokensusage.total_cached_tokens
Shared across organizations?No; isolated per workspace on the Claude APINoThe documentation read here does not say

Where the three agree: stable content first, variable content last; an exact prefix; caching saves processing, not window space. Two documented details are easy to miss. Claude's documentation says a cache entry becomes available only after the first response starts, so parallel requests sent at the same moment do not hit each other's cache. Gemini's documentation says that for Gemini 4 models in multi-turn loops, the full previous request is cached only when the new request adds at least 2,050 tokens.

What to log per call, next to the manifest: cached tokens read, cached tokens written, uncached input tokens, and a hash of the fixed prefix. When writes stay high call after call, the hash tells you whether the prefix is changing (a timestamp, a reordered tool list) or the cache is expiring between calls.

Security note. A cache is shared within its isolation boundary. Keep per-tenant or per-user data out of any prefix you expect to be reused across tenants, and put tenant-specific instructions after the shared part. The example's system instructions name tenant-a; in a multi-tenant deployment that line would split the cache per tenant, which is the correct behavior, and it should be a conscious choice.

For a Principal engineer: the trade-offs that do not go awayBigger windows, caching against compaction, and who owns the budget

Large windows change the failure, not the need. A 1M-token window removes the hard edge for most runs and keeps context rot, token cost and the square-law growth of re-sent history. The budget question moves from "will it fit" to "is it worth reading", which is harder to test because nothing fails.

Caching and compaction pull in opposite directions. Caching rewards a history that only grows at the end; compaction rewrites it. Compact rarely and substantially (a clear_at_least floor, a trigger well below the limit), at natural boundaries such as the end of a subtask, so that each rebuild buys many cheap calls afterwards.

The budget is a product decision with an owner. Shares encode what the business considers decisive evidence. A change to them should go through the same review as a change to the eligibility rules, with every case rerun and the manifests compared. In the example the shares are part of context_version; a change produces ctx-2026.12, not an edit in place.

Count with the provider, budget with yourself. Your estimator decides the packing; the provider's counter decides the bill and the limit. Keep a margin, log both, and alert when they drift apart.

A checklist for a window budget#

CheckWhy
The budget is split into shares per kind of content, and shares do not lend by defaultOne large item must not take the room of the decisive one (UW-202)
Every trimmable source has its decisive fields at the head, or is extracted instead of trimmedA trim keeps the head, whatever the head holds
A trimmed item carries a visible mark and kept/total tokens in the manifestA silent trim looks like a complete item
Utilization is tracked per context versionTokens that no answer uses cost money and attention
Total input tokens per run are logged, not only the last windowA run's cost is the sum of its windows
Tool definitions and system instructions come first, byte-identical across callsThey are the cache prefix; a reorder or a timestamp breaks it
Variable data (submission, time, run id) comes after the stable prefixAnything per call in the prefix makes every call a cache miss
Cache read and write tokens are logged per callA falling hit rate means the prefix changed or expired

In one breath#

A window budget is only useful when it is divided: give each kind of content its own share, so that a long tool output can lose its tail but never push out the rule the answer depends on. Truncation keeps the head and destroys the tail silently, so mark every trim and check where each source puts its decisive fields. A bigger window is not the fix, because irrelevant tokens dilute attention, recall degrades with length, and a run pays for every window it re-sends. Put the stable part of the window first and keep it byte-identical, so the provider can cache it.

What comes next#

You can now divide a window and keep its prefix cacheable. But the run UW-203 used above keeps growing: 40 steps, 4,541 tokens of history, and the early steps hold the roof conflict, the rule and an open question to the broker. No share can hold all of it forever. The next page, Compaction, is about shortening a run's history so it can continue, and about the test that proves the facts a later step needs are still there.