Routing and load balancing

Why instances are not interchangeable

Ordinary web load balancing often assumes every backend instance is interchangeable — stateless, equally capable of handling any request, so routing decisions can be as simple as round-robin or least-connections. Inference instances break this assumption in several real ways: one instance might already have a request’s relevant prompt prefix cached, per prefix caching, while another has nothing cached at all; one might be nearly saturated while another sits mostly idle; and in a disaggregated serving deployment, instances are not even running the same job — some handle prefill exclusively, others handle decode exclusively. Routing that ignores all of this performs noticeably worse than routing that accounts for it.

What a smarter router actually considers

A router built for inference specifically can use several engine-level signals a generic load balancer never sees: current queue depth at each instance, to avoid piling more work onto an already-saturated one; cache state, to prefer an instance that already has a relevant prefix cached even if it means slightly more queueing, since the cache hit can save more time than the extra wait costs; and, in a disaggregated deployment, which phase a given instance actually serves, so a prefill request is never mistakenly sent to a decode-only instance. None of these signals exist in a typical stateless web service, which is exactly why inference routing needs its own dedicated approach rather than reusing generic load-balancing tooling unchanged.

What to watch for

A recent conference talk from OpenAI practitioners directly on building routing policy from engine-level signals — load, cache state — for production inference, which is precisely the kind of concrete, signal-driven routing this lesson covers.

The specific case of cache-aware routing

Worth dwelling on directly, because it is the least intuitive of these signals: routing a request to a busier instance over an idle one can be the faster choice, specifically when the busier instance already has the request’s prompt prefix cached. The idle instance would have to compute that entire prefix from scratch, while the busy one only has to process whatever is genuinely new, even after accounting for some additional queueing delay. This is a direct, practical consequence of everything prefix caching makes possible, and it is exactly the kind of routing decision a generic, cache-unaware load balancer has no way to make correctly.

What happens when routing gets it wrong

A poorly tuned router produces two visible failure patterns worth recognising. Hot-spotting happens when too many requests land on the same instance — because it has a popular cached prefix, say — while other instances sit comparatively idle, which looks like healthy average utilisation on a dashboard while individual users experience very uneven latency depending on which instance they happened to land on. Thrashing happens when a router switches an instance’s workload too aggressively, constantly swapping which model or adapter is loaded in response to short-term signals, paying the real cost of a reload or cache eviction more often than the routing decision actually saves. Both are symptoms of a router optimising too narrowly for one signal — cache affinity alone, or queue depth alone — rather than balancing several signals together, which is the real engineering difficulty in building a good one.

Routing for adapters, and routing at scale

Where multiple fine-tuned variants of a base model — covered in Upshard’s Machine Learning and AI Engineer roadmaps as LoRA adapters — are served off shared infrastructure, routing also has to account for which adapter is currently loaded on which instance, since switching an instance’s active adapter has its own cost. The deeper links cover the Kubernetes-specific tooling built around exactly these signals — queue depth, cache affinity, adapter availability — for teams running inference at a scale where manual routing decisions are no longer practical to make by hand.