Cost and latency

The asymmetry everything follows from

Input and output tokens are not priced alike, and the reason is mechanical rather than purely commercial, following directly from the prefill-and-decode split covered in how LLMs work. The prompt is processed in a single parallel pass across every position at once. Every output token, by contrast, requires its own separate sequential forward pass through the entire network, attending over everything generated before it — simply far more computational work per token produced than per token merely read. A typical pricing split has output tokens costing somewhere between three and five times as much per token as input tokens, directly reflecting this real difference in underlying compute cost. Two practical consequences follow. Long prompts are cheaper than they intuitively feel, so a generously detailed prompt costs less than the same volume of generated output would — though stuffing context still costs accuracy for the reasons covered in the context window lesson, independent of the dollar cost. And verbose output is genuinely expensive, which makes an explicit instruction toward conciseness a real cost control measure, not merely a stylistic preference.

A worked end-to-end example

Consider a feature handling ten thousand requests a day, each averaging two thousand input tokens and five hundred output tokens, against a hypothetical price of three dollars per million input tokens and fifteen dollars per million output tokens. Input cost per day comes to ten thousand requests times two thousand tokens, divided by a million, times three dollars — sixty dollars. Output cost comes to ten thousand times five hundred, divided by a million, times fifteen dollars — seventy-five dollars. Total: a hundred and thirty-five dollars a day, close to four thousand dollars a month, for a feature that might feel small in a demo. Doubling the input through a longer system prompt or more retrieved context roughly doubles the input half of that bill; doubling the average output length through verbose, unconcise responses roughly doubles the larger output half — which is exactly why trimming output length is usually the higher-leverage lever of the two for a text-heavy feature.

The unit that matters: cost per completed task

The number worth optimising is cost per completed task, not cost per token or even cost per call. An agent that costs a fraction as much per individual call but needs four attempts to actually succeed at the task has not saved anything at all relative to a more expensive agent that reliably succeeds on its first attempt — it may well have cost more overall, while also taking longer. This is the same reasoning that makes evals essential to any genuine cost comparison between two candidate approaches: a cheaper model that needs substantially more retries or more elaborate scaffolding to reach the same success rate is not actually a cheaper system once that overhead is honestly counted.

Levers, roughly in order of typical return

Caching, covered in its own lesson, is usually the largest and easiest single win available, arranging prompts so the stable portion comes first and can be served from cache on repeat calls. Routing sends most requests — which are, in most real systems, genuinely easy — to a small, cheap model, escalating only the requests that actually need a more capable one; the escalation decision is itself a classification problem that a small model can often make reliably on its own. Shortening output through structured responses rather than unconstrained prose, with no restated question and no unnecessary preamble before the substantive answer, directly attacks the more expensive half of the pricing asymmetry. Batching, where latency is not user-facing — overnight processing, bulk classification jobs, large backfills — trades a longer turnaround time for a substantial, often significant, provider discount on bulk asynchronous processing. Model cascades formalise the routing idea further: try the cheapest model first, check its own confidence or validate its output against some cheap criterion, and escalate to a progressively more capable and more expensive model only when that check fails, rather than deciding up front which single model every request will use.

Little’s Law, applied to concurrent requests

Little’s Law — see the full formula — states that the average number of requests concurrently in a system equals the average arrival rate multiplied by the average time each request spends in the system. Applied directly to an LLM service, this means a service receiving ten requests per second, each taking an average of three seconds to complete from start to finish, has on average thirty requests in flight at any given moment — not ten. This directly determines the concurrency a serving infrastructure must actually be provisioned to handle at once, and it is also exactly why reducing a feature’s per-request latency, even without changing the arrival rate at all, directly reduces the infrastructure capacity needed to serve that same traffic.

Time to first token, and what streaming actually fixes

What a user perceives as “speed” is dominated far more by time to first token — how long an interface sits visibly empty before anything appears — than by the total time the complete response takes to finish generating. Time to first token is dominated by the prefill phase, so it grows directly with prompt length, which is yet another reason caching matters here specifically: a cache hit skips the bulk of prefill and the first visible token can appear almost immediately afterward. Streaming the response token by token as it is generated is the standard remedy, and the honest framing is that it changes a user’s perception of elapsed time rather than the actual total duration — the total generation time is completely unchanged by streaming, only the long, empty wait before anything visible happens disappears. In an agent making several sequential tool calls before producing any visible output at all, this compounds in a way streaming cannot help with: ten sequential tool calls at two seconds each is twenty full seconds before the first useful word can even begin streaming, since there is simply nothing yet to stream. Parallelising independent tool calls and reducing the total number of sequential iterations an agent needs matters considerably more there than anything to do with the model’s own raw generation speed.