Skip to content

Pattern

How to control the cost of AI agents

You control AI agent costs by sending routine work to small, cheap models and saving large ones for hard steps, caching repeated prompt content, and putting hard budgets and caps on every key, agent and job. Clients can then pay providers directly on their own keys or use a managed allowance.

StudiedLast reviewed 4 min read

Explained from public sources. Mai has not built this yet.

On this page

In plain English

An agent pays for every thought. Each time it reads, plans or writes, it calls a language model and is charged for the text going in and coming out. Most of those calls are simple, such as sorting an email or pulling a date from a document, and do not need the most powerful model. Cost control means matching the model to the job, not paying twice for the same reading, and putting hard limits in place so a mistake cannot quietly run up a bill.

How it works

Four levers, used together.

Route to the smallest model that does the job. A routing step classifies each request or each step of a job and picks a model for it. Classification, extraction and short replies go to small, fast models; multi-step reasoning, tricky drafting and anything the small model is unsure of goes to a larger one. This can be a simple rule (“the triage step always uses the small model”), or a learned router. The RouteLLM research from 2024 trained routers on human preference data to choose between a strong and a weak model per query, and reported cutting cost by more than half in some settings without a drop in answer quality. Either way, the choice should sit in configuration, not be hard-coded into each agent.

Triage, Drafting, Planning. Then Model gateway. Then Small model, Large model. Then Cost per job.

A gateway routes each call, applies caching and enforces budgets

Cache what repeats. Agents send the same long preamble on every call: instructions, tool definitions, reference documents. Prompt caching lets the provider reuse its processing of an identical prefix. Anthropic’s version, for instance, caches the tools, system prompt and earlier messages up to a marked point; later requests with exactly the same prefix are billed at a fraction of the normal input rate, while writing to the cache costs a little more than normal and entries expire after a few minutes unless reused. To benefit, put stable content first and changing content last, and keep that stable part truly unchanged.

Budgets and caps. Limits belong in the infrastructure, where the agent cannot argue with them. A model gateway sits between agents and providers and can enforce them centrally. LiteLLM, for example, supports spending limits per virtual key, user and team, each with an optional reset period, plus token and request rate limits, and rejects calls once a budget is spent. Alongside that, each job should carry its own cap on steps and spend, so one confused agent stops itself early.

Bring your own key, or a managed allowance. Someone has to hold the provider account. With bring your own key, the system uses the client’s own provider accounts, the provider bills the client directly, and the client sees and controls usage in their own dashboard. With a managed allowance, the builder holds the accounts and charges for usage within an agreed monthly amount. Routers such as OpenRouter support both models, letting a client’s own provider keys sit behind the same interface. The trade-off is control and transparency against convenience.

Where it goes wrong

  • Runaway loops. An agent that keeps retrying a failing tool, or two agents handing work back and forth, spends money with every turn. Step limits and per-job caps are the guard.
  • Big model everywhere. Defaulting every call to the most capable model is the most common overspend, and often no better for simple steps.
  • Small model everywhere. The opposite mistake: a cheap model that gets the hard step wrong costs more in rework and human time than it saved. Routing decisions should be checked with tests, not assumed.
  • Caches that never hit. Putting a timestamp or the user’s name at the top of the prompt changes the prefix on every call and defeats caching.
  • Budgets that fail open. A spending limit that is only checked when the tracking database is available may let everything through when it is not. LiteLLM’s own documentation notes that its budgets need a database to be enforced, and offers a setting to refuse requests rather than admit them when limits cannot be checked.
  • Context bloat. Long-running conversations that keep every past message grow more expensive on every call. Summarising, or storing facts in agent memory instead of the transcript, keeps calls lean.
  • Measuring the wrong thing. Cost per call says little. Cost per completed task, read from the traces, shows which agent or step is worth optimising.

How Mai AI uses it

In a Mai AI system the agents never talk to a model provider directly. They call a gateway that decides which model handles each step, applies caching, and enforces budgets per agent and per client, with each job also carrying its own step and spend cap. In the AI chief of staff built for a UK property group, models sit behind a LiteLLM gateway so a model can be swapped without touching the agents, which is the same design that makes routing and budgets possible.

I design each system so the client can choose: their own model keys, paying the provider directly with no model charge from me, or a managed allowance that I look after within an agreed limit. The process page sets out how each option is priced.

Questions and answers

Why are AI agents expensive to run?

Agents call a model many times per task, and each call resends the instructions, tool descriptions and conversation so far. Costs grow with the number of steps and the size of that context, and a loop that never ends keeps spending.

How do you reduce AI agent costs?

Route simple steps to small models, cache the parts of the prompt that repeat, keep context short, and cap spend per key, per agent and per job. Measure cost per completed task, not per call, so you can see where the money goes.

What does bring your own key mean for AI systems?

It means the system uses the client's own account with the model provider, so usage is billed by the provider directly to the client. The alternative is a managed allowance, where the builder holds the provider accounts and charges for usage within an agreed limit.

Can I set a spending limit on an AI agent?

Yes. Model gateways such as LiteLLM can enforce budgets per key, user or team that reset on a schedule, and reject requests once the budget is used. A per-job cap in the agent's own code adds a second line of defence against runaway loops.

Sources

  1. LiteLLM documentation, budgets and rate limitsdocs.litellm.ai
  2. Anthropic documentation, Prompt cachingplatform.claude.com
  3. Ong et al., RouteLLM, Learning to route LLMs with preference data (arXiv)arxiv.org
  4. OpenRouter documentation, Bring your own keyopenrouter.ai