Understanding AI Costs: Tokens, Context, Caching

As long as AI is a €20-a-month subscription, the cost question is boring. It gets interesting once companies build models into processes via API – then the bill scales with every call, and teams get their first surprise invoice. The good news: the cost mechanics are simple, and so are the levers.

Key takeaways

• You pay per token – input (prompt plus context) and output (the answer), with output usually costing noticeably more.
• Between a provider’s strongest and cheapest model, price typically differs by one to two orders of magnitude.
• The biggest hidden cost driver is context sent along with every call – chat histories and whole documents that get paid for again with every request.
• Model routing, curating context, and prompt caching are the three most effective levers, in exactly that order.
• Measuring cost per case rather than just reading the monthly total catches outliers before they become a problem.

The mechanics: three pricing factors

Three factors determine the size of any AI bill: the number of tokens, the model class, and the size of the context sent along with the request. Understand these three and you understand the entire cost logic behind every API call – the rest is arithmetic.

Tokens: You’re billed per processed chunk of text – input (everything you send, including system instructions and documents) and output (the answer), with output costing noticeably more. Model class: Between a provider’s strongest and cheapest model, price typically differs by one to two orders of magnitude – the single most important decision of all. Context size: Because models have no memory, the entire relevant context gets sent along – and paid for – with every call: the chat history, the attached documents, the knowledge-base excerpts. This is exactly where most surprises come from.

Worked example: what an automation case actually costs

A simple worked example shows how much model choice matters – often more than every other lever combined.

A support team has 10,000 incoming requests a month automatically pre-classified. Each request brings in an average of 300 input tokens (text plus system prompt) and around 50 output tokens (category plus a short note) – 3.5 million processed tokens a month in total. Assume the cheap model costs around €20 a month for this total load. The top-tier model from the same provider – at the typical one-to-two-order-of-magnitude price difference – would come in at €200 to €2,000, for a result that’s barely measurably better on a pure classification task. The exact amounts depend on the provider and the current price, but the order of magnitude of the difference is typical: for standard tasks, model choice is by far the biggest cost lever – not caching, not batch processing.

The five levers, ranked by impact

1. Model routing: Not every task needs the top-tier model. Classification, extraction and standard copy run almost as well on cheap models – the routing principle from our article on Local AI vs. cloud is first and foremost a cost principle.
2. Curate context: Relevant passages instead of whole documents, capped chat histories, lean system prompts. Side effect: precise context also improves quality – context engineering and cost discipline are the same exercise.
3. Prompt caching: Recurring context (system instructions, knowledge bases) can be cached with most providers – identical portions then cost only a fraction. Often the single biggest lever for applications with fixed templates.
4. Batch instead of real time: Where answers aren’t needed instantly (overnight runs, bulk processing), providers offer significant discounts for batched processing.
5. Limit output: Tight target formats (“answer as a table, maximum X rows”) save on the most expensive token type – and force more precise results as a side effect.

Model classes compared

Not every task needs the same model class – this rough breakdown helps before you even pick a specific provider.

Model class	Typical relative price	Typical tasks	Quality bar
Budget class	Base (1×)	Classification, extraction, formatting	Low–medium
Mid class	Roughly 5–10×	Summarisation, standard copy, simple analysis	Medium
Top class	Roughly 20–100×	Complex reasoning, agents, code, edge cases	High

Rule of thumb: classify the task first, then choose the model class – not the other way round. If you want to test whether a budget class is enough, check on twenty to fifty real examples whether the quality suffices for the specific case, rather than relying on gut feeling. A common middle path: set the mid class as the default for everything that doesn’t clearly fall into the budget category, and escalate to the top class only on demonstrated need – not assumed need. That keeps the bill manageable without having to run a fresh cost-benefit analysis for every new task.

The most common cost mistakes

Top-tier model for standard tasks: Out of convenience, everything runs through the most expensive available model. Fix: introduce model routing by task type.

Endless chat histories: A conversation gets carried on for weeks instead of restarted – the entire history is paid for again with every message. Fix: deliberately end conversations when the topic changes.

Whole documents instead of passages: Entire PDFs get sent with every call, even though only one paragraph is relevant. Fix: curate context instead of piling it up.

Caching not enabled: Recurring system prompts get paid for in full on every call, even though caching is available. Fix: actively check for and use the provider’s caching option.

Only reading the monthly total: Without a breakdown by use case, it stays unclear which process is driving costs. Fix: track cost per case, not just the total bill.

The management view

Treat AI costs like cloud costs: measure cost per case (not just the monthly total), set budgets and alerts, and run the simple sum for every new use case – what does the call cost, what does the case save? For multi-step AI agents that run several steps in sequence for one task, the context effect multiplies further – each step reprocesses the history so far, which is why agent workflows benefit especially from curated context and caching. Most productive AI use cases pay off handsomely; you only spot the exceptions if you look.

Subscription or API: what pays off when

For individuals and small teams with irregular use, a subscription is almost always the right choice; for automated processes with predictable volume, switching to the API pays off. The decision hinges on three questions, not on gut feeling.

When a subscription: Usage is manual, irregular, and spread across several task types – one chat window for everything. The fixed monthly price is an advantage here, because it stays predictable regardless of actual consumption.

When the API: A task repeats automatically (classifying, summarising, extracting), runs in the background without a manual chat window, and can be captured in templates. Here, tighter control over model choice, context and caching pays off – costs that would stay invisible in a flat-rate subscription become directly visible here, and therefore directly reducible.

The mixed case: Most growing teams start with a subscription and gradually move individual, clearly defined processes over to the API once a use case proves stable and recurring – not the whole company at once.

The bottom line

AI rarely gets expensive because of the price per token. It gets expensive because nobody did the maths: the wrong model for a simple task, the conversation that never ended, the whole document instead of the relevant passage. Anyone who knows the three pricing factors and pulls the five levers in the right order – model routing first, then context, then caching – usually cuts the bill by a multiple, without losing quality. The next concrete step: log cost per case instead of just the monthly total for one week, and the biggest lever will reveal itself.