Part 4 of 6 · 5 min read
Interview questions and limits
Covers: Interview questions, with the follow-ups; Common mistakes; What the example does not do.
Interview questions, with the follow-ups
How do you reduce the cost of an agent that re-sends a long history every step?
"I use the provider's prompt cache and I shape the request for it: stable content first (system prompt, tools, standing state), history append-only, and the new turn last, so each step reads the old prefix from the cache and pays full price only for what is new. I keep anything that changes per step, like a timestamp, out of the prefix. In my simulation that took the cost of a 40-step run to 15 percent of the uncached cost, and a timestamp in the system prompt made it 125 percent." Follow-up: "What if the history gets too long?" I reduce it, but in batches: compaction edits old messages, which misses the cache from the edit onward, so compacting every step cost twice as much as leaving the history alone even though the window was smaller.
Does compaction save money or cost money?
"It depends on the cache discount, and I'd measure it. Compaction shrinks every later request but breaks the prefix at the edit. With a 90 percent read discount, compacting every step was more expensive than append-only; with a 50 percent discount it was cheaper. Compacting in batches was cheapest at every price I tried, because it keeps most steps append-only." Follow-up: "Where would you set the threshold?" Where the cost of a long window plus the accuracy loss from length (module 6) exceeds the cost of one miss; I'd compute it from my own prices and traffic.
Why did the cache stop working?
"I'd look in this order: something volatile near the top (a timestamp, an id, a reordered tool list); an edit to an earlier message; a gap longer than the TTL; a prefix below the minimum cacheable size; or a change to the model, tool definitions or system prompt, which is a different prefix. Then I'd check the provider's cached-token count per call, which tells me directly." Follow-up: "Is a cache hit a correctness risk?" No: it reuses computation for identical input, so the output distribution is the same; the risks are cost and latency, and privacy if a cache were shared across tenants, which is a provider question.
Common mistakes
- A timestamp, request id or user name in the system prompt.
- Reordering tools or sections between requests.
- Compacting or summarising on every step.
- Rewriting the notes or state block every step.
- Quoting one provider's discount as universal. Terms differ and change.
- Ignoring idle time in low-traffic agents.
What the example does not do
- No real provider. Real caches have minimum sizes, breakpoints, per-provider quirks and storage fees; mine has a minimum size, a TTL and two multipliers.
- Abstract content. Segments are identities and token counts, so tokenisation effects (a small edit changing token boundaries) are not modelled.
- One agent shape. Parallel sub-agents sharing a prefix, or a shared cache across users, behave differently (module 8 and your provider's documentation).
- Optional real-provider experiment (I did not run this): make the same 40 steps with a real API and compare the reported cached-token counts for append-only and compact-every-step.
Check yourself
You add the current time to the first line of the system prompt so the model knows the date. What happens to your cache hits? (They drop to zero. The cache needs an identical prefix, and the first changed token ends the hit. Put the time in the newest message instead.)
For the curious Compaction can cost more than it saves
Compacting every step shrank the window by 43 percent in the example but cost almost twice as much as leaving the history alone, because every rewrite broke the cached prefix. Compacting in batches was cheapest. If the cache read price is high (0.5 of normal), the answer flips, so the result depends on your provider price.
What you will use daily at work
| Practice | What it looks like |
|---|---|
| Stable first, volatile last | System prompt, tools, notes, history, then the new turn. |
| Watch idle time | An entry that expires between requests turns the next request into a full write. |
| Read your provider cache fields | Most responses report cached and uncached tokens separately. |
Draft: confirm your provider caching rules: minimum size, time-to-live, breakpoints and prices.
What you must be able to say (one page)
- A hit needs an identical prefix. The first changed token ends the hit; everything after it is recomputed, and on most providers written to the cache again at a premium.
- Therefore order the request from most stable to most volatile: system prompt, tool definitions, standing notes, history, then the new turn. Never put a timestamp, a request id or any per-step value near the top.
- Quote these numbers, with the caveat that the prices are my parameters (read at 0.1 of the normal input price, write at 1.25, time-to-live of 5 steps): over 40 steps, append-only with caching cost 0.15 of no caching; a timestamp in the header cost 1.25, worse than no cache; compacting every step cost 0.29, almost twice append-only even though the window was 43 percent smaller; compacting in batches cost 0.14, the cheapest.
- The trade-off flips with the discount: at a read price of 0.5, compacting every step (0.44) beats append-only (0.54). So "compaction saves money" or "compaction costs money" are both wrong without the price.
- Idle time matters: an entry that expires between requests turns the next request into a full-price write.