Latency versus throughput

Comparing systems the right way

It is tempting to compare two serving systems, or two configurations of the same system, by their raw throughput numbers — whichever produces more tokens per second wins. This is a poor comparison on its own, because throughput can always be bought by tolerating worse latency: push enough concurrent requests through a system and its aggregate throughput climbs, while any individual request waits longer and longer for its turn. The fair comparison fixes a latency target first — say, a p95 time to first token under two seconds — and then asks how much throughput each system or configuration can sustain while staying inside that target.

What to watch for

The video teaches exactly this methodology: pick a latency target first, then read off the throughput the system can sustain at that target, and shows the queuing curve where latency “blows up” once the system is pushed past its sustainable capacity. This reframing — latency target first, throughput as the derived number — is the single most useful habit in this lesson.

Why latency explodes rather than degrading gracefully

A serving system has a finite capacity to process requests. Below that capacity, adding a few more concurrent requests raises latency only slightly, since there is spare capacity to absorb them. Once incoming demand exceeds what the system can actually process, a queue starts to form, and it keeps growing for as long as demand exceeds capacity — each new request waits behind an ever-lengthening line of requests ahead of it. This is why the latency curve, plotted against increasing load, stays nearly flat for a while and then rises almost vertically: the system has crossed from “keeping up” to “falling behind,” and those two regimes look completely different.

Little’s Law describes this precisely: the average number of requests in the system equals the arrival rate multiplied by the average time each one spends there. Once arrivals outpace the rate at which requests complete, that average time spent in the system — which includes however long a request waits in queue — grows without bound.

Reading a latency-throughput curve in practice

A real benchmark of this tradeoff typically produces a curve rather than a single number: throughput plotted on one axis, the latency required to achieve it on the other. Near the left side of that curve, small increases in throughput cost very little extra latency — there is spare capacity being used efficiently. Further along, each additional unit of throughput starts costing noticeably more latency, and past a certain point the curve bends sharply upward, which is the queueing effect described above taking hold. Reading this curve for a specific system tells you directly where the genuinely efficient operating region ends — and operating anywhere past that bend means accepting a disproportionate latency cost for comparatively little additional throughput, which is rarely the right trade once you can see the curve laid out this way.

Tuning is choosing a point, not maximising a number

A production serving system is almost never configured for maximum possible throughput, because that configuration accepts whatever latency results, however bad. It is configured for the highest throughput achievable while staying inside a latency target the product actually needs — a real, deliberate choice specific to what is being built, not a universal setting to copy from somewhere else. A latency-sensitive interactive product and a throughput-sensitive batch pipeline will reasonably land on very different points along this same curve for the exact same hardware and the exact same model.