Keeping an AI feature affordable in production

A feature ships, people use it, and the first full month of invoices is several times the estimate. Nothing is broken. The answers are good, the latency is acceptable, and the only thing wrong is the number at the bottom of the bill. Keeping an AI feature affordable in production turns out to be bookkeeping more than engineering: knowing what each call costs, and refusing to pay for tokens nobody reads.

Where the money actually goes

A model call is billed on what you send plus what comes back, and in most production features that ratio is lopsided. Thirty or forty tokens go in for every token that comes out. The expensive part is not the clever answer, it is the text attached to every request without anyone thinking about it.

Four things do most of the damage:

1. The system prompt. Two thousand tokens of instructions, rules and examples, sent on every call. At two hundred thousand calls a month that is four hundred million input tokens before a single user types a word.
2. Retrieved context. A search returns twenty passages and the code sends all twenty, because sending more felt safer than sending less. Four of them contain the answer. The other sixteen are paid for and ignored.
3. Conversation history. Turn ten resends turns one through nine. The cost of a conversation grows with roughly the square of its length, so the tail of a long chat can cost more than the first twenty turns put together.
4. Retries. A retry resends the whole input. A client that tries three times on a rate limit turns one expensive call into three, and a pool of workers all retrying together turns a bad minute into an expensive hour. This is the same failure I described in treating a rate limit as a permanent failure, except here it also shows up on the invoice.

Output tokens are priced higher each, but there are usually far fewer of them. The exception is a call with no output limit, where a model that decides to be thorough writes three pages that the interface truncates anyway.

Measure before you change anything

Every guess I have made about where the cost sits has been wrong at least once, so I no longer optimise before there is a table to look at. One row per model call, written after the response, never sampled:

await costLog.insert({
  requestId,                       // same id as the HTTP request
  feature: 'ticket_classify',      // not 'ai', the feature name
  tier: 'small',
  attempt,                         // 1 for the first try
  inputTokens: usage.input,
  cachedInputTokens: usage.cachedInput,
  outputTokens: usage.output,
  costMicros: priceMicros(tier, usage),
  accountId,
  latencyMs
});

Writing cost in millionths of a unit keeps it an integer, which means you can sum a million rows without floating point drift. Once a week of data exists, one query tells you where to spend your attention:

SELECT feature,
       count(*)                                  AS calls,
       sum(cost_micros) / 1e6                    AS total,
       sum(cost_micros) / count(*) / 1e6         AS per_call,
       sum(input_tokens) / sum(output_tokens)    AS in_out_ratio,
       sum(cached_input_tokens) * 100
         / nullif(sum(input_tokens), 0)          AS cached_pct
FROM model_calls
WHERE created_at >= now() - interval '7 days'
GROUP BY feature
ORDER BY total DESC;

The first time I ran the equivalent of that query on a live feature, the top line was a background job that nobody had opened since the week it shipped. It was re-summarising the same records every night because it had no idea what it had already done.

The fix

Five changes, in the order that usually pays best.

Freeze the prefix and cache it. Everything stable goes first and stays byte for byte identical: the system prompt, the tool definitions, the examples. Everything that changes goes after it. A cache read is billed at a fraction of a fresh input token, so the savings are large, but only while the prefix never moves.

const messages = [
  { role: 'system', content: SYSTEM_PROMPT, cacheable: true },  // stable
  { role: 'system', content: TOOL_SCHEMA,   cacheable: true },  // stable
  ...history,                                                   // changes
  { role: 'user',   content: question }
];

A single dynamic value in front of that block, a timestamp or the user name, sets the hit rate to zero and nobody notices until the bill arrives.

Trim the retrieved context. Rank the passages, then cut. Take the top four instead of the top twenty and measure whether anyone can tell. In one system I worked on the answer quality did not move at all between eight passages and three, and the input dropped by roughly two thirds.

Route by task, not by product. Classification, routing, extraction and short rewrites go to a small model. The large model is reserved for the calls where a reader would notice the difference: the final draft, the summary a human will send. Decide this with a blind comparison on fifty real inputs, not with an opinion.

Batch what is not urgent. Anything the user is not waiting for, nightly digests, backfills, bulk classification, goes through a queue and a batched endpoint, which is typically priced at about half. The same queue also protects the interactive path from being starved, which is the argument I made for moving a hundred thousand pasted rows into a queue.

Cap the spend before the call. A per account, per day counter, incremented with the estimated cost of the call, checked before anything leaves the process:

const key = `spend:${accountId}:${utcDay()}`;
const spent = await counters.increment(key, estimate.micros, { ttlSeconds: 172800 });

if (spent > limits.dailyMicros(accountId)) {
  await counters.increment(key, -estimate.micros);   // give the reservation back
  return degraded(question);                         // cheaper path, honest message
}

After the response, adjust the counter by the difference between the estimate and the real usage. The estimate only has to be close, because its job is to stop a runaway loop, not to be an invoice.

How to check it worked

Run the same query a week later and put the two outputs side by side. The shape of the change is more convincing than any single number:

feature           calls     total   per_call   cached_pct
ticket_classify  184,320     41.20   0.000224           91
reply_draft       12,940     96.10   0.007427           88
weekly_digest        620     18.40   0.029677            0

Three things to read in that table. The classification feature has high volume and low unit cost, which is what routing to a small model is supposed to look like. The cached percentage near ninety means the prefix is stable. The digest job has a cached percentage of zero, which is the next thing to look at, because a nightly job is exactly the kind of work that should be batched and cached.

What to watch out for

• A cache write costs more than a plain call. Caching a prefix that gets used once makes that request more expensive, not less. Cache the prefix that is shared by thousands of calls, not the one that is unique per user.
• Averages hide the tail. One account with a two hundred turn conversation can outspend ten thousand ordinary requests. Always look at the ninety fifth percentile of cost per request, not only the mean.
• Your token count is an estimate, not the bill. Reconcile the sum of your logged cost against the actual invoice once a month. If they drift apart, your price table is out of date.
• A cheaper model that needs two attempts and a longer prompt is not cheaper. Count the full cost of the task, including the validation pass you added to make the small model safe.

The lesson that survives every one of these features is that cost is a property of the payload, not of the model you picked. A prompt that grew by four hundred tokens during a month of small improvements is a price increase that nobody approved, and the only way to see it is to have logged the number before and after. Put the cost of a request next to its latency in the same table, treat both as things that can regress, and the bill stops being a monthly surprise. The habit is the one that turns a slow query into the right index: find the expensive thing first, change one thing, then look at the number again.