You deploy an AI agent to production by treating everything after the demo as the real work: an evaluation harness that proves it meets an outcome agreed up front, guardrails and permissions around its tools, observability on every run, a cost-per-task budget, and a human in the loop where a mistake would be expensive. A demo runs once; production runs the same workflow thousands of times, and that is where unmeasured agents fail.
The demo-to-production gap
Every agent project reaches the same seductive moment: the demo works. The agent researched the thing, drafted the thing, called the right tools, and the room nodded. Most projects then attempt the shortest path to production — wire the demo to real data, add a login, ship it — and most of those agents are quietly switched off within months.
The reason is that a demo and a production system answer different questions. The demo answers “can this work?” — once, on inputs someone chose, with the author watching. Production answers “does this work every day?” — on inputs nobody curated, at a volume where every weakness is multiplied, with nobody watching unless the system itself raises a hand. The gap between those questions is not closed by a better model or a better prompt. It is closed by engineering, and the engineering has a checklist.
The deployment checklist
Six things, in the order they usually bite.
1. An evaluation harness, with the measure agreed first. Before an agent ships, there must be an answer to “what number says this is working?” — task success on real, representative data, scored repeatably. Agree the measure up front, build a harness that runs the agent against it on every change, and refuse to ship on the strength of good anecdotes. An agent should arrive in production measured, not asserted — the fuller argument is in our piece on agent evaluation.
2. Guardrails and permissions on every tool. Production tools are typed and validated — schemas on inputs and outputs, so malformed calls are caught at the boundary rather than passed downstream. Permissions follow least privilege: read-only wherever read-only does the job, scoped credentials, and hard rules enforced by the system rather than requested in the prompt. An agent’s blast radius is set by what its tools may touch.
3. Observability from day one. Every run traced end to end: each retrieval, each model call, each tool invocation, each validation, with cost and latency per step. When something fails — and something will — the trace is the difference between reading the evidence and holding a meeting. What to monitor in a production agent covers the five signals that carry the operational weight.
4. A cost-per-task budget. A demo costs pennies because it runs once; production multiplies every inefficiency by thousands of runs a month. Set a cost budget per task at design time, architect to it — model routing, context discipline, caching, capped loops — and track it in production like any other KPI. Rising cost with stable quality is still a regression. The levers are in our piece on agent cost optimisation.
5. A human in the loop, placed deliberately. Find the decisions where a mistake would be expensive — sending, committing, spending, anything customer-facing — and put review there. Let the agent run unattended everywhere else. The design failure at both extremes is the same: review everything and the agent saves nothing; review nothing and the first bad output is a production incident.
6. Failure paths and rollback. Timeouts, retries that are idempotent and narrow, fallbacks when a source is down, a defined behaviour when the agent is out of its depth — designed before launch, not discovered after. And because agents sit on moving foundations (models deprecate, APIs change shape), a deployment needs the boring virtues too: version-pinned behaviour, a way to reproduce yesterday’s run, and a switch that turns the agent off without turning the workflow off.
What the first month in production looks like
The first weeks after deployment are where the checklist pays for itself, because production immediately starts telling you things the demo could not. Real inputs arrive that no one curated, and some of them find the edges: a document format the retrieval handles badly, a tool that times out at a frequency the tests never showed, a class of request the agent should be refusing and is not. None of this means the deployment failed — it means the system is finally being measured against reality.
The operating rhythm is simple. Read the traces on the failures, not just the failure counts — the categories matter more than the totals, because different failure modes have different fixes. Watch cost per task against the budget from day one; early drift is far cheaper to correct than established drift. Feed the awkward real-world cases back into the evaluation set, so the harness grows teeth as the agent meets the world. And resist the temptation to tune by anecdote: a stakeholder forwarding one bad output is a data point for the eval set, not a mandate to re-prompt the agent that afternoon. The teams that struggle in month one are almost never short of information — they shipped without the instrumentation to read it.
Deployment is the start of Run, not the end of Build
The checklist gets an agent safely into production. What keeps it there is operational: the evaluation keeps running as the agent changes, the traces keep being read, the cost keeps being watched, and someone owns the number the agent was built to move. Agents drift — the data underneath them shifts, usage patterns change, a model update alters behaviour at the margins — and unwatched drift is how a working agent becomes a switched-off one.
This is why we structure every engagement as Scope, Build, Run rather than ending at delivery. Scope agrees the measure and proves the agent is worth building; Build ships it with the checklist above in from day one; Run keeps it evaluated, observed, and improving in production — reported against the number, every week it is live. Both of the agents we run in production work this way: the tender-intelligence agent for Magellan Circle and our own compliance-first outreach engine carry the same instrumentation, and it is why they are still running rather than still being demoed.
If you have an agent that works in a demo and a business that needs it to work every day, the checklist above is the distance between the two. Book a 30-minute call and we will walk your case through it.