Cost and latency

Where the time goes

A response has two phases. Processing the input (often called prefill) happens largely in parallel and shows up as time to first token. Generating the output (decode) is sequential: one token after another. For most chat-style requests the output dominates, so the strongest levers are:

• ask for shorter answers and set an output token limit that fits the task,
• stream, so users read while the rest is generated,
• run independent calls in parallel rather than one after another,
• use a smaller model where quality allows.

Token budgets

Decide a budget per feature: how many input tokens (system prompt, history, retrieved passages) and output tokens a request may use. Enforce it in code: trim or summarise old conversation turns, cap the number of retrieved chunks, and cap tool rounds in agent loops. Also limit per user and per tenant, since one script calling your endpoint in a loop can run up a large bill. Providers enforce their own rate limits in requests and tokens per minute; handle 429 responses with backoff.

Prompt caching

When many requests start with the same content, such as a long system prompt, tool definitions or a reference document, providers can reuse the work done on that prefix. Cached input tokens are billed at a reduced rate and processed faster. The details differ: some providers cache automatically, others ask you to mark cache breakpoints; some charge a little extra for the request that writes the cache; entries expire after a period of disuse (at the time of writing, defaults range from a few minutes to about half an hour, with longer options on some APIs); and there is a minimum prompt length below which nothing is cached. Check your provider's page, and confirm hits in the usage numbers each response reports. What is common is the rule that the cache matches a prefix: put stable content first, and keep it identical. A timestamp or user name near the top of the system prompt defeats it.

Separately, you can cache whole answers in your own application (keyed by a hash of model, prompt and settings) when identical requests are common and the answer may be reused.

Batching and model routing

Work that nobody is waiting for (nightly summaries, backfills, evaluation runs) can go through a provider's batch interface: you submit many requests and collect results later, typically at a discount and with a completion window measured in hours. Without one, use a queue with a fixed number of workers so you stay within rate limits.

Not every step needs the largest model. Classification, routing, extraction from short text and simple rewrites often work well on smaller, faster, cheaper models; keep the large model for hard reasoning or long, nuanced writing. Decide with your evaluation set, not by assumption: run each candidate model on it and compare quality, cost and latency together.

The calculator uses made-up relative prices (units per thousand tokens), not real ones. Change the numbers to your provider's current rates to see what caching and routing do to a feature's cost.

Watch the totals, not just the averages

Track cost and latency per feature and per tenant, with percentiles (p50, p95, p99), not only averages: long answers, large documents and agent loops create a heavy tail. Alert on sudden changes, which often mean a prompt change made answers longer or a loop stopped terminating.