Part 4 of 6 · 2 min read
Provider layer and break-even
Treat the local model as one more adapter, and work out when owning hardware beats paying per request.
Write an AdapterLocal with the same complete(request) -> Response method as module 7: translate the neutral request to the local payload, translate the reply back, and map LocalError kinds to ProviderError kinds. unreachable, timeout, dropped and loading are retryable (server). not_loaded is a bad request. Then a router can prefer the local model for private or easy requests and fall back to a hosted API, or the reverse.
Local does not mean free of work
Prompts tuned for a hosted model often need changes for a smaller local one: shorter instructions, stricter structured-output handling (module 3's retry loop matters more), and lower expectations for tool calling. Re-run your test set before trusting the swap.
Break-even
breakeven_requests_per_month(fixed_monthly=400, api_cost_per_request=0.01) # 40,000
If a machine costs 400 dollars a month (rent or depreciation plus power) and each API request would cost one cent, you need about 40,000 requests a month before owning it is cheaper, ignoring your time and any quality difference. If a local request has its own cost that is not lower, there is no break-even at all (the function returns infinity). Use real prices and your measured throughput before deciding. Also check that one machine can actually serve 40,000 requests a month at the speed you need.
What you will use daily at work
| Practice | What it looks like |
|---|---|
| A local runtime for experiments | Developers run a small model locally to test code without spending money or sending data out. |
| A shared inference server | For real traffic, a team runs a serving system on GPUs and measures throughput and latency under load (Level 13). |
| Private-data routing | Requests with sensitive data go to the local or self-hosted model, the rest to a hosted one. |
| Re-testing on every model change | A new model or quantisation level must pass the same prompt tests. |
Draft: confirm current runtimes, model formats and hardware guidance before relying on them.