Deploying and scaling agents

From a working script to a running service

An agent that works when you run it from the command line on your own machine has cleared a low bar: one request, one user, no competition for resources, and a failure just means you read the stack trace and try again. Production deployment asks a completely different question — can this same agent serve many users at once, recover from a failed step without a human watching, keep its cost per request from running away, and be updated without taking the whole thing down? None of that is implied by “it works,” and all of it is usually invisible until the first time real traffic hits the thing.

A worked example: what actually changes

Picture a local script: it loads a system prompt, enters a loop that calls an LLM, executes whatever tool call comes back, feeds the result back in, and exits when the agent produces a final answer. It stores conversation state in a plain Python list in memory. It calls external APIs directly and prints errors to the console if something fails.

Turning this into a service that can actually be deployed changes nearly every one of those assumptions:

• Concurrency. The in-memory list keyed to one running process has to become per-request or per-session state, usually in an external store (a database, a cache), because multiple requests are now running through the same deployed instance at once and can’t share one list without corrupting each other’s conversations.
• Retries and failure handling. A tool call that fails midway can no longer just print an error — it needs a retry policy (with backoff, and a cap so a persistently failing call doesn’t loop forever) and a defined fallback behavior, because nobody is watching the console to notice and restart it by hand.
• Observability. As covered in the tracing lesson, a service running unattended needs its traces and metrics captured automatically, since a production failure with no trace is a failure nobody can diagnose.
• A deployment pipeline. Instead of running python agent.py on a laptop, the agent gets packaged — typically as a container — and deployed through a managed agent platform or infrastructure-as-code (so the exact environment, dependencies, and configuration are reproducible), with health checks that restart it automatically if it crashes and a path to roll out an updated version without downtime.

None of these changes touch the agent’s actual reasoning or tool logic — the part that was “the agent” to begin with. They’re entirely about the difference between a script that runs once when you launch it and a service that has to keep running correctly under conditions nobody is watching in real time.

Cost control: prompt caching

Every agent call typically resends a large, mostly-unchanging prefix: the system prompt describing the agent’s role, the full set of tool schemas it can call, and often a block of retrieved context. Without caching, the model reprocesses every token of that prefix on every single call, even though it’s often identical call to call within a session or even across sessions. Prompt caching lets the provider store the already-computed representation of that unchanged prefix and reuse it on subsequent calls, billing the cached portion at a steep discount (and serving it faster) compared to processing it fresh. For an agent that might send the same several-thousand-token system prompt and tool schema set on every one of a dozen steps in a single run, this is often the single largest cost lever available — a cost reduction that costs nothing in terms of changing the agent’s actual behavior.

Cost control: routing sub-tasks to cheaper models

Not every step in an agent’s loop needs the most capable model available. Classifying which of five tools to call, summarizing a short tool result, or extracting a field from structured output are all tasks a smaller, cheaper model handles about as well as the frontier model the agent might use for its hardest reasoning step. Model routing assigns each sub-task to the cheapest model that can do it reliably, reserving the expensive model for steps that actually need its capability — the complex planning step, the final synthesis where mistakes are costly. An agent that routes its five easy sub-tasks to a model costing a fraction as much per token, while keeping its one hard reasoning step on the frontier model, can cut its total per-run cost substantially without any measurable drop in the quality of its final output, simply because most of an agent’s steps were never the ones that needed the expensive model in the first place.