Continuous batching

The waste static batching cannot avoid

The simplest way to batch requests together is static batching: collect a fixed group of requests, run them all through decode together step by step, and only start the next batch once every request in the current one has finished generating. The problem is that real requests rarely finish at the same time — one might generate a short answer in a dozen tokens while another runs for hundreds. Under static batching, the GPU slots freed up by the shorter requests sit idle for the remainder of the batch, waiting on whichever request happens to run longest, which wastes exactly the capacity batching was meant to use efficiently in the first place.

Continuous batching: add and remove at the step level

Continuous batching removes this waste by operating at a finer granularity: rather than waiting for an entire batch to finish together, it adds a newly arrived request into the running batch and removes a finished one the moment it completes, at the level of individual decode steps. A slot freed by a request that just finished is immediately available for the next waiting request, rather than sitting idle until the whole batch clears. This keeps the GPU doing useful work far more consistently under real traffic, where request lengths vary unpredictably and arrive at unpredictable times rather than as a conveniently uniform group.

What to watch for

An AI21 Labs inference engineer gives a focused talk specifically on how continuous batching, alongside PagedAttention, keeps GPUs busy under real, bursty traffic — directly on this lesson’s scope, including the practical engineering of making it work well rather than only the conceptual argument for why it helps.

Why this matters more as traffic gets more realistic

The gap between static and continuous batching grows with how varied real traffic actually is. A workload where every request happens to be nearly identical in length sees little benefit from continuous batching, since static batching’s waste is small when nothing finishes early. Real production traffic is essentially never this uniform — request lengths vary by user, by task, by how much context is attached — which is exactly why continuous batching has become close to a baseline expectation in any serious inference engine rather than an optional optimisation, and why every engine covered in inference engines implements some form of it.

Connecting back to the throughput-latency tradeoff

Continuous batching does not eliminate the fundamental latency-versus-throughput tradeoff covered earlier in this roadmap — a batch that is too large still slows down any individual request’s per-token speed, exactly as before. What it changes is efficiency at a given batch size: for the same number of concurrent requests in flight, continuous batching extracts substantially more useful throughput from the hardware than static batching would, by never leaving a freed slot idle. Little’s Law still governs how many requests a system can hold in flight at once; continuous batching is what makes sure that capacity is actually used rather than wasted.