Context Budget¶

A million-token context window is not free real estate. Every token in it is paid for on every iteration, adds latency, and dilutes the model's attention. A context budget treats the window as a finite resource and allocates it deliberately.

Kroki

Why budget matters¶

There are three reasons to keep the context small:

Cost. Tokens in the input are billed on every turn of an agent loop. A long prefix multiplied by ten tool calls becomes expensive fast.
Speed. Prefill time grows with the input length. The user feels the delay before the first output token.
Quality. Models do not attend to all tokens equally. Facts in the middle of a long prompt are recalled less reliably.

Budget allocation¶

Divide the context window into protected slices and spend them in order:

Slice	Priority	Why
System prompt	Highest	Defines identity, rules, tools. Never truncate or reorder it mid-session.
Working / enriched query	High	Current task and user intent.
RAG results	Medium	Only include filtered, relevant, compressed results.
Chat history	Medium–Low	Drop oldest first, but keep tool-call/result pairs so the agent does not lose state.
Reserve for response	Mandatory	Leave enough headroom for the model to think and answer.

If the model runs out of room while generating, it will cut off mid-sentence or emit only the reasoning trace with no final answer. A common failure mode is to maximize input context and forget that the model also needs output tokens.

Compaction rules¶

When the window is too full, apply compaction in order of cost:

Drop or summarize old user/assistant turns. Keep tool calls and results; they are the agent's ground truth.
Summarize long tool results. Keep the first preview plus a path to the full output.
Rebuild the whole context once. Do not mutate it incrementally — middle mutations invalidate the KV cache.
Auto-compact as a last resort. Use a background model to summarize the entire history, with a failure-rate circuit breaker to avoid loops.

Budget-aware context compression¶

For agents that run many turns, treat compression as a sequential decision under a budget, not a one-time summarization step. ContextBudget (BACM) conditions each turn's policy on remaining context headroom and uses commit-block aggregation to choose among three regimes [Wu et al., 2026]:

Budget pressure	Regime	Behavior
High	Preserve	Keep the full interaction history; defer compression
Moderate	Selective	Compress redundant segments while keeping salient evidence
Low	Collapse	Fully aggregate the context to stay within limits

Training with a progressively tightened budget curriculum yielded >1.6× gains over strong baselines in high-complexity tasks and a 5× improvement in a 32-objective QA setting. The takeaway: adaptive, budget-conditioned compression outperforms fixed-turn summarization.

Thinking strategy¶

Frontier models expose reasoning or "extended thinking" modes. These modes emit a hidden chain of thought before the visible answer. The cost is often 4–5× higher than a normal response because reasoning tokens are billed as output.

Not every request needs deep thought:

Request type	Reasoning
"How much is a latte?"	Off. Lookup or simple classification.
"Plan a quarterly retro order from team history."	On. Requires analysis and trade-offs.
Tool execution after a plan is known	Off. The agent is following an already-decided plan.

A practical policy:

Enable thinking for the first one or two turns while the agent is analyzing the task and building a plan.
Disable thinking for execution turns where the agent is invoking already-selected tools.
Let a cheap classifier decide whether the current request merits reasoning effort.

Providers expose this as reasoning.effort, budget_tokens, thinking_level, or thinking_token_budget. Treat the reasoning budget as a cap, not a target, and instrument the budget-exhaustion rate as a quality metric.

Failure modes and metrics¶

Failure	Cause	Metric to watch
Output starvation	Input context consumes almost the whole window	Available output headroom in tokens
Context thrashing	Auto-compact runs repeatedly and fails	Compaction failure rate, loop counter
Over-compression	Relaxed budget but agent summarizes away evidence	Task F1 / answer accuracy vs. uncompressed baseline
Under-compression	Tight budget but agent keeps too much	Overflow rate, truncation rate
KV-cache invalidation	Mid-context edits after prefix was cached	TTFT spike, cost per turn
Reasoning-budget exhaustion	`thinking` tokens exceed cap	Budget-exhaustion rate, final-answer completeness

Set per-slice budgets and alarm on budget-exhaustion and cache-hit rates.

Prompt caching and compression¶

Prompt caching reuses KV states for repeated prefixes, turning a per-turn re-read into a cheap cache hit. On a multi-turn deep-research benchmark across OpenAI, Anthropic, and Google, caching reduced API costs by 41–80% and TTFT by 13–31% [Lumer et al., 2026]. Strategy matters:

System-prompt-only caching is the most consistent across cost and latency.
Full-context caching that includes dynamic tool results can increase latency because the cache is rewritten every turn.
Exclude tool results and place dynamic content at the end of the prompt for the cleanest wins.

Cache-Aware Prompt Compression (CAPC) adds a second lever. Anthropic Sonnet 4.6's cache is two-tier: a hot tier below ~3,500 tokens with a plateaued hit rate of ~0.83, and a persistent tier above it. CAPC pairs query-agnostic compression with cache_control markers and a tier-preserving ratio bound, cutting cost by 49% over cache-only, 64% over query-aware compression, and 90% over vanilla while keeping quality within 0.05 of the uncompressed baseline [Song, 2026].

Design principles¶

Every slice has a budget. System, history, RAG, documents, and output room are all bounded.
Never mutate the middle. Reassemble once instead of editing the context every turn.
Reasoning is a switch, not a default. Turn it on only when the request genuinely benefits from it.
Treat caching and compression as one cost system. A query-aware compressor that changes the prefix on every call can destroy cache savings.

Deeper dive: context as a managed resource¶

Context engineering reframes the agent problem from "write the right prompt" to "curate the right tokens at every step of a multi-turn run." Anthropic's 2025 definition treats it as the set of strategies for selecting, maintaining, and evicting information during inference [Digital Applied, 2026]. In production, the context window behaves more like a managed resource than free storage: every token adds prefill cost and latency, competes for the model's attention, and contributes to context rot — the measured drop in output quality as input length grows. This degradation appears across every major model family and is not cured by million-token windows; it is only delayed [Digital Applied, 2026].

A token budget should therefore be treated as an architecture constraint, not a tuning parameter. Tian Pan proposes dividing the budget by phase: planning consumes 10–30%, execution 40–60%, verification can dominate in frameworks that explicitly check their work, and output generation needs a protected 10–20% reserve [Pan, 2026]. Reasoning or extended-thinking tokens are part of the planning and verification budget, so they should be routed through the same reallocation and degradation tiers as any other token. When reality diverges from the plan — for example, a tool call returns far more data than expected — the budget must be reallocated dynamically. Useful tactics include complexity-aware estimates before execution, progressive compaction at milestones, sub-agent isolation that returns only condensed summaries, and explicit degradation tiers that drop verification, planning, or final detail rather than silently truncating [Pan, 2026].

This architectural view maps onto the four failure modes and four remediation levers Digital Applied identifies: context poisoning, distraction, confusion, and clash, matched to write, select, compress, and isolate [Digital Applied, 2026]. No single technique is sufficient; production runs usually need all four layered, with the dominant lever shifting as turns accumulate. Anthropic's published evaluations make the case concrete: context editing plus a memory tool improved performance by up to 39% and reduced token consumption by 84% in a 100-turn web-search task, while subagent isolation delivered more than a 90% improvement over a single-agent baseline on a research benchmark [Digital Applied, 2026].

A practical phase-level budget ladder for long-horizon runs is:

Phase	Typical share	What to do when it runs over
Planning & reasoning	10–30%	Escalate only for tasks that need decomposition or trade-offs
Execution & tool results	40–60%	Pre-filter tool lists, preview result sizes, or delegate to subagents
Verification	variable; up to ~66% in explicit frameworks	Replace the LLM verifier with a cheaper checker or skip the self-check
Output reserve	10–20%	Protect unless the run explicitly enters a truncation tier

Use these shares as alarm thresholds and per-slice ceilings, not as one-size-fits-all targets.

References¶

OpenAI. "Reasoning with the Responses API." https://platform.openai.com/docs/guides/reasoning
Anthropic. "Extended Thinking Guide." https://docs.anthropic.com/en/docs/build-with-claude/extended-thinking
Wu, Y., et al. "ContextBudget: Budget-Aware Context Management for Long-Horizon Search Agents." arXiv, 2026. https://arxiv.org/abs/2604.01664
Lumer, E., et al. "Don't Break the Cache: An Evaluation of Prompt Caching for Long-Horizon Agentic Tasks." arXiv, 2026. https://arxiv.org/abs/2601.06007
Song, Y. "Cache-Aware Prompt Compression: A Two-Tier Cost Model for LLM API Caching." arXiv, 2026. https://arxiv.org/abs/2607.15516
Digital Applied. "Context Engineering: Agent Reliability Playbook 2026." Digital Applied blog, 2026. https://www.digitalapplied.com/blog/context-engineering-agent-reliability-playbook-2026
Pan, T. "Token Budget as Architecture Constraint: Designing Agents That Work Under Hard Ceilings." tianpan.co, 2026. https://tianpan.co/blog/2026-04-13-token-budget-as-architecture-constraint