How do you reduce the running cost of an AI agent?

You reduce an agent's running cost through architecture, not bargaining: route each step to the smallest model that can do it, cut the context each call carries, cache what repeats (prompts, retrieved passages, tool schemas), and design loops and retries so failures don't silently multiply calls. Most of an agent's cost per task is fixed by these design decisions before the first invoice arrives.

Why do AI agent costs explode at scale?

Because a demo runs once and production runs thousands of times a day, so every inefficiency is multiplied. The usual causes are one frontier model doing every step including trivial ones, context that re-sends the same material on every call, retry logic that re-reads everything on each pass, and nobody watching cost per task — so the growth is only noticed on the invoice.

What is cost per task and why does it matter?

Cost per task is the total model, tool, and infrastructure spend divided by completed units of work — per document processed, per lead researched, per ticket resolved. It matters because it is the number that decides whether the agent's business case survives scaling. An agent that saves an hour of work but costs more than the hour is not automation; it is an expensive habit.

Should every step of an agent use the most capable model?

No. Most agent workflows mix a few steps that genuinely need frontier reasoning with many that don't — classification, extraction, formatting, routine drafting. Routing those to smaller, cheaper models typically cuts cost several-fold with no measurable quality loss, provided an eval harness confirms the smaller model clears the same bar.

When should you start managing an agent's cost?

At scoping, before the build. Agree a cost-per-task budget alongside the quality KPI, design to it, and then track it continuously in production — rising cost is a regression even when the answers still look fine. Retro-fitting cost discipline after launch usually means re-architecting, which is far more expensive than designing to a budget from the start.

AI agent cost optimisation: how to control cost per task at scale

You control an AI agent’s running cost at design time, not at spend time. Most of what an agent costs per task is set by architecture — which model handles which step, how much context each call carries, what gets cached, how loops and retries behave — long before the first invoice arrives. Dashboards and spend alerts are lagging indicators: they tell you what you spent, not that it was avoidable.

This is the part of the business case a demo never shows. A demo costs pennies because it runs once. Production runs the same workflow thousands of times a month, and every inefficiency is multiplied by that volume. An agent that looks cheap in a five-minute demo can eat its own business case at ten thousand runs a month — and by then the architecture that decided the cost is already built.

Where the money actually goes

Agent spend concentrates in a handful of places, and they compound:

One model doing everything. The frontier reasoning model that plans the workflow is also formatting dates, classifying inputs, and writing boilerplate — work a model a tenth of the price does just as well.
Context that never shrinks. Every call re-sends the full conversation, the full document, the full tool catalogue. Token cost scales with what you carry, not with what the step needs.
Loops and retries that multiply silently. A retry that re-reads the entire context, an agent that re-fetches what it already holds, a planning loop that runs three times when once would do. Each run is individually within budget; the pattern is not.
Output nobody trimmed. Verbose answers cost tokens on the way out too, and downstream steps then carry that verbosity as input.

The same task, on the same model, commonly runs several-fold apart in cost between two teams — the difference is nothing but design.

The levers, in order of impact

Model routing. Split the workflow into steps and give each step the smallest model that clears the quality bar. Frontier reasoning where judgement genuinely lives; small, fast models for extraction, classification, and formatting. This is usually the single largest saving, and an eval harness is what makes it safe — the smaller model has to prove it clears the same bar, not just look fine.

Context discipline. Send each call what it needs and nothing else. Summarise long histories, retrieve passages instead of whole documents, prune the tool catalogue to the tools the step can actually use. Context is the quiet majority of most agents’ token bills.

Caching. Prompt caching for the static preamble every call shares; retrieval caching for material the workflow reads repeatedly; memoising tool results that don’t change between steps. Cached tokens are billed at a fraction of fresh ones — and the preamble is often most of the prompt.

Loop and retry design. Retries should be idempotent and narrow — re-run the failed step with the context that step needs, not the whole run from the top. Cap loop iterations. Make an agent check what it already knows before it fetches again.

A cost gate in the trace. Give every run a cost budget and check it the way you check an eval score. A run that blows its budget is a failure to investigate, even when the output looks right — because at volume, “right but triple the cost” is a regression.

Cost per task is a KPI, not a line item

The discipline that ties the levers together: agree a cost-per-task budget at scoping, alongside the quality measure, and design to it. Then track it in production continuously — per run, per step, against the budget, traced the way you trace everything else — so drift shows up in the trace, not on the invoice. Rising cost with stable quality is still a regression; something in the architecture has started spending more to do the same work. Treat it exactly as you would a falling eval score, because economically it is one. This is the same principle as evaluating an agent — cost per task is one of the four signals an evaluation weighs, and it is the one that decides whether the agent scales.

Our point of view

At Agent Foundry Labs, cost engineering ships in every build — model routing, caching, and token discipline designed to a cost-per-task budget, then tracked in production like any other KPI. We ran this discipline on our own outreach engine, a compliance-first research agent we build and run in-house, and engineered its running cost down by roughly 60–70% — same task, same quality bar, measured before and after. None of it required a cheaper model provider; all of it was architecture.

There is a simple test for any agent you are running or buying: can you say what it costs per completed task, what the budget is, and how you would know if that number started climbing? If not, that gap is worth closing before you scale — because by the time cost reaches the expense sheet, the architecture has already decided most of it. Book a 30-minute call and we will start with the number.

Where the money actually goes

The levers, in order of impact

Cost per task is a KPI, not a line item

Our point of view

Quick answers