Prompt caching: Cost tracking and prompt caching

The provider reuses the start of an identical prompt at a lower price, so the order of your prompt decides your bill.

Prompt caching

Prefix caching

The provider stores the processed form of the start of your prompt and reuses it when a later request begins with exactly the same tokens. Cached input tokens are usually billed at a fraction of the normal rate and may also arrive faster. Some providers charge extra the first time, to write the entry.

Rules that hold in the general case (check your provider for the exact numbers):

Only an identical beginning matches. One different token early on means nothing after it can be reused.
Minimum size. Very short prompts are not cached.
Block alignment. Matching often happens in fixed-size blocks, so the last partial block is not reused.
Entries expire, and using an entry usually refreshes it.

Hence the layout rule: stable first, changing last.

good:  [system instructions][tool definitions][long document][conversation so far][new question]
bad:   [current time][system instructions][tool definitions][long document] ...

The same words in two orders, 30 simulated requests. Real output of the cache simulation.

In the simulation, putting a timestamp first gives a cache hit share of 0% and a total of $0.66. Putting it last gives 96% and $0.16. Nothing about the content changed.

Things that silently break the cache

A timestamp or request ID near the top; tool definitions listed in a different order each time; a user's name in the system prompt; editing an earlier message in the history. Keep the top of the prompt byte-for-byte identical across calls.

Using the simulation

cache = PrefixCache(min_tokens=1024, block=128, ttl=300)
r = cache.request(tokens, now=12.0)   # r.cached_tokens, r.written_tokens, r.input_tokens
led.add("run", "user", "large", step, Usage(r.input_tokens, 200, r.cached_tokens, r.written_tokens))

Real providers either cache automatically or ask you to mark where the cacheable part ends. Either way, the cost formula is the one above, fed with the cached and written counts from the response.

Check yourself

Your system prompt starts with "Today is Tuesday 14:32". Why is that expensive? (The first tokens change every minute, so no later token can match a stored prefix. Move the time to the end, or to the user message.)

Other ways to cut the bill

A smaller model for easy steps (routing). Classify first, then choose, and measure that quality holds.
Shorter history: trim or summarise (module 1), at the price of losing detail.
Shorter outputs: ask for concise answers, or set a max_tokens cap that fits the task.
Batch or off-peak options, where a provider offers them, for work that can wait.
Do not call at all: cache your own results for repeated identical questions.

Measure before and after each change using the ledger, never by feel.

What you will use daily at work

Practice	What it looks like
Usage logged on every call	The cost record is written next to the request log, with run, user, model and prompt version.
Dashboards and alerts	Spend per user, per feature and per day, with an alert when a day is far above normal.
Budget caps per run and per user	The agent loop checks both before each call.
Cache-friendly prompts	A review rule: nothing that changes goes near the top of a prompt.

Draft: confirm token fields, cache pricing, minimum sizes and expiry against your provider's current documentation before relying on them.