Cost and the ledger: Cost tracking and prompt caching

How a call's cost is computed from its token usage, and a log that answers who spent what.

Usage

The token counts a provider returns with each response: tokens in, tokens out and, where supported, how many input tokens were served from cache or written to cache. Always read these from the response, because your own estimate can differ from what you are billed for.

Cost is a weighted sum, priced per million tokens:

def cost_of(u, p):
    fresh = u.input_tokens - u.cached_tokens - u.cache_write_tokens
    return (fresh * p.input_per_m + u.cached_tokens * cached_rate
            + u.cache_write_tokens * write_rate + u.output_tokens * p.output_per_m) / 1e6

By hand, with the placeholder "large" prices: 2,300 tokens in and 150 out cost 2300 x 3 / 1e6 + 150 x 15 / 1e6 = $0.00915. Output tokens cost more per token, but agents usually send far more input than they receive, so input often dominates the bill.

A ten-step run with 900 new tokens of history per step. Real output of the example code, placeholder prices.

The picture is the key idea of the module. Step 1 costs $0.00825 and step 10 costs $0.03255, about four times as much, and the run total ($0.204) is a sum, not ten times one call.

Check yourself

A call has 4,000 input and 300 output tokens on the "small" model (0.25 and 1.25 dollars per million). What does it cost? ($0.001375: 4000 x 0.25 = 1000, 300 x 1.25 = 375, divided by a million.)

The ledger

Ledger

An append-only log with one record per model call: which run, which user, which model, which step, the usage and the dollars. Everything else (totals, per-user bills, alerts) is a query over it.

led = Ledger()
led.add("run-7", "asha", "large", step=1, usage=Usage(1000, 100))
led.total(run_id="run-7")        # cost of one run
led.group_by("user")             # who is spending the most, largest first
led.save("ledger.jsonl")         # one JSON object per line; Ledger.load() reads it back

Three habits make a ledger useful:

Tag at write time. You cannot add the run ID later. Record run, user, model and step on every call.
Store tokens as well as dollars. Prices change. With tokens stored you can re-price history.
Reconcile with the invoice. reconcile(your_total, invoice_total) reports the gap. A gap above a few percent means calls you are not logging (retries, background jobs) or a wrong price.

Count retries

A retried call is billed whenever the provider processed it, even if you never saw the reply. Log each attempt (module 5's retry_call reports them) or your totals will run low.

For the curious A bug the first version had

cost_of checked for impossible usage (more cached tokens than input tokens) with an assert. Python removes asserts when run with python -O, and then 100 input tokens with 80 cached and 40 written produced a negative number of fresh tokens and a silently wrong price. It is now a real ValueError, and negative token counts are refused too. Rule: validate money inputs with exceptions, never with assert.