Covers: Results over 40 steps; A practical layout.
Results over 40 steps
Setup: system prompt 2,000 tokens, tool definitions 1,500, each step adds a 150-token assistant turn and a 600-token tool output. Compaction (when used) clears old tool outputs to a 25-token stub when the window passes 12,000 tokens. Cost is in units of an uncached input token.
Policy
Relative cost (no cache = 1.00)
Share of tokens read from cache
Mean window
no cache, keep all
1.00
0%
18,955
cache, append only
0.15
95%
18,955
cache, timestamp in the system prompt
1.25
0%
18,955
cache, compact every step
0.29
64%
10,804
cache, compact in batches (to half the budget)
0.14
85%
9,971
cache, idle gap longer than the TTL before every fifth step
0.39
74%
18,955
Cost by policy, and how the three history strategies compare as the cached-read price rises.
Reading it:
Caching is the big lever (1.00 to 0.15) and it needs nothing but a stable prefix.
A volatile header is worse than having no cache, because every request pays the write premium and never gets a read. This is the single most common mistake.
Compacting every step shrinks the window by 43 percent and costs almost twice as much as leaving the history alone. Each compaction edits an old message, so the cache hits only the part before the edit.
Batch compaction wins (0.14): compaction happens rarely, to half the budget, so most steps are append-only again. It beats append-only slightly because the window is half the size.
Idle gaps cost real money. With a gap longer than the TTL before every fifth step, the cost rises from 0.15 to 0.39; with a TTL of 15 steps it falls back to 0.15.
The price sweep.
Cached-read price
0.10
0.25
0.50
1.00 (no discount)
append only
0.15
0.30
0.54
1.01
compact every step
0.29
0.35
0.44
0.62
compact in batches
0.14
0.21
0.32
0.54
With a deep discount, leave the history alone; as the discount shrinks, shrinking the history pays; batching is never worse than the append-only history in this setup. This is the quantitative form of the rule to start with one agent.
A practical layout
[ system prompt ] stable for weeks
[ tool definitions ] stable per release, fixed order
[ standing notes / state ] changes rarely; update in batches
[ history ... ] append-only
[ new user turn ] the only uncached part
Put per-step values in the last segment, not the system prompt (time, user id, request id).
Do not rewrite the state block every step; change it when it must change, and expect one miss from that point.
Compact on a threshold and to a target well below it, so the next compaction is many steps away.
Keep the traffic warm if the cost of a miss is high and the TTL is short, or accept the miss for sporadic work.
Watch the minimum size. Below the provider's minimum cacheable length, nothing is cached.
Measure. Providers report cached and uncached input tokens per call; chart the read share per step (tokens per step is the number to watch).