The Hidden Cost of AI Features: Token Bills, Latency, and How to Control Both

The prototype cost almost nothing to run. Three dollars over a fortnight of testing, which felt like a rounding error and got mentioned in the launch meeting as a reason not to worry.

Six weeks after launch the monthly bill was a real line item, and nobody could explain why it had grown so much faster than user numbers.

This is the normal shape of AI API costs, and it catches out experienced teams. The reason is that the prototype and the production system are charged for different things, and the difference is not obvious until it is on an invoice.

What you are actually paying for

You are billed by tokens — roughly, pieces of words — in both directions. Input tokens are everything you send: the system prompt, retrieved context, conversation history, the user's message. Output tokens are what comes back. Output is typically several times more expensive per token than input.

That pricing shape has a consequence most teams miss: your bill is driven mostly by what you send, not by what the user types.

A user asking "what are your opening hours?" sends eight tokens. If your system wraps that in a two-thousand-token instruction block, four retrieved documents, and the last ten messages of conversation, you paid for roughly four thousand tokens to answer an eight-token question.

Multiply by every message, by every user.

The four things that inflate the bill

Context length that grows silently

Every improvement to quality tends to add context. Someone adds examples to the system prompt because the output was inconsistent. Someone increases retrieved chunks from three to eight because an answer was missed. Someone extends conversation history because the assistant forgot something.

Each change is defensible and each raises the per-message cost permanently. Six months later nobody remembers why the prompt is three thousand tokens, and nobody wants to be the person who removes something and makes quality worse.

Retries you cannot see

When a response fails validation — malformed structure, a refused answer, a timeout — most implementations retry. You pay for the failed attempt as well as the successful one.

A five percent failure rate with two retries is not a five percent cost increase; it is meaningfully worse, and it is invisible unless you log attempts rather than successes.

Users who are chattier than you modelled

Prototype testing involves single questions. Real users have conversations. And because history accumulates, the tenth message in a conversation costs far more than the first — you are resending everything that came before.

A twenty-message conversation can cost several times what twenty separate questions would, for the same amount of user value.

Nobody looking at the number

The most expensive factor. Inference spend is usage-based and compounds quietly. Teams that check monthly discover problems a month late, by which point the habit is baked into the product.

The levers that actually work

In rough order of payoff per hour of engineering effort.

Trim the context aggressively

The highest-return work, and the least glamorous. Read your actual prompt. Not the version in the design document — the real assembled string being sent.

Most production prompts we examine contain instructions that no longer apply, examples that duplicate each other, and retrieved chunks that are merely topical rather than relevant. Cutting this typically reduces cost and, surprisingly often, improves answers, because the genuinely relevant passage is no longer buried.

Summarise conversation history instead of resending it

Rather than sending every previous message, keep a short running summary and the last couple of exchanges. Quality barely moves for most applications, and the cost of a long conversation stops growing linearly.

Route easy work to a cheaper model

Not every request needs your most capable model. Classification, intent detection, routing, and simple extraction are handled well by smaller and much cheaper models.

A two-stage design — cheap model decides what kind of request this is, capable model handles it only if needed — often cuts spend substantially while improving latency on the simple paths.

Cache what repeats

Two kinds of caching matter. Many providers support caching a stable prompt prefix so you are not charged full price for resending the same system instructions every time; if your prompt has a large fixed portion, this is close to free money. Separately, cache complete answers to genuinely common questions — a meaningful share of support queries are the same handful of questions, and those do not need a model at all.

Rate limit from day one

Per-user and per-session limits protect you from both accidents and abuse. An unauthenticated endpoint that calls a model without limits is a bill waiting to happen.

If your numbers are drifting

If your spend is growing faster than your usage and you cannot explain the gap, that is usually a context problem rather than a traffic problem — and it is fixable in days rather than weeks.

If you would like someone to look at what your system is actually sending, we are happy to go through it with you. It is usually a short piece of work with a disproportionate payoff.

Latency is the same problem wearing different clothes

Everything that inflates your bill also slows you down, because both scale with tokens.

Long context takes longer to process. Long outputs take longer to generate — output length is usually the dominant factor in how long a response takes. Retries multiply wait time. So trimming for cost improves speed at the same time, which is an unusually pleasant trade.

Two things help specifically with perceived speed:

Stream the response. Words appearing immediately feel fast even when total time is unchanged. Not streaming is the most common self-inflicted latency problem we see.

Ask for shorter answers. If your assistant returns four paragraphs where two would do, you are paying twice and the user is waiting twice as long. Constrain output length in the prompt and enforce it.

For a sense of how much the model choice itself matters here, provider documentation such as Anthropic's model overview lays out the trade-offs between capability, speed and price per model, which is the comparison worth making before optimising anything else.

What to measure

Five numbers, tracked daily rather than monthly:

• Total spend, plotted against active users, so you see the ratio rather than the total
• Average input and output tokens per request, which is where drift shows first
• Requests per active user per day
• Retry rate, logged as attempts rather than successes
• The 95th percentile response time, not the average

If cost per active user is rising while the product has not changed, something in your context assembly has grown. That ratio is the single most useful alarm you can set.

What it should actually cost

For most small and mid-size products, inference lands between $30 and $400 per month. Heavy document processing or high chat volume can reach $1,000 to $3,000 per month.

If you are materially above that without corresponding volume, the cause is almost always context length rather than traffic — which is good news, because it is the cheapest thing on the list to fix.

For planning a build: a support assistant grounded in your content runs $2,500 to $7,000 for us, a document extraction pipeline $4,000 to $12,000, and an in-app copilot with tool access $8,000 to $25,000. Those sit below typical Western agency pricing because of how our team is structured, not because anything is skipped — and we would always rather scope the running cost with you before the build than after.

Where AI API costs hide in the architecture

Beyond the prompt itself, three architectural choices quietly determine your bill. They are worth settling early, because changing them later means reworking the feature.

Synchronous where asynchronous would do. If a user submits fifty documents for processing, doing that inline means holding a connection open, timing out, and retrying — paying twice. A queue with a notification when the work completes costs less and fails better. Batch processing is also frequently discounted by providers, which makes the asynchronous path cheaper twice over.

Regenerating instead of storing. We regularly find systems that re-summarise the same document every time someone opens it. The summary has not changed. Store the output alongside the source, invalidate it when the source changes, and the cost of that feature drops to near zero after first use.

No ceiling on user-triggered work. Any feature where one user action can trigger many model calls — "analyse my entire workspace", "rewrite all of these" — needs an explicit cap. Without one, a single enthusiastic customer can generate a month's budget in an afternoon, and you will only find out afterwards.

The pattern across all three is the same: model calls should be treated like any other expensive external operation. You would not hit a paid third-party API in an unbounded loop without thinking about it; the same discipline applies here, and the OWASP guidance on unrestricted resource consumption is a reasonable framing for why it is a security concern as well as a financial one.

Build the cost model before the feature

The teams that avoid this problem do one thing differently: they estimate the per-message cost during design, multiply by expected usage, and check whether the number is acceptable before writing code.

It takes twenty minutes. It surfaces the cases where a feature is not viable at the price point you had in mind, which is far better learned before you build it than after you launch it.

If you are planning an AI feature and want realistic numbers for what it will cost to run — not just to build — book a free consultation call or email nitheesh@playxoft.com. We will model it with you first, because a feature that is unaffordable at scale is not a feature.