The inference stack

What you'll learn

A model checkpoint on disk is not a running service; several distinct layers sit between it and an API a user can call, and each layer is where a different category of performance work happens.

The inference engine handles the model-specific heavy lifting — batching, memory management, the actual forward pass — while the serving framework around it handles HTTP, routing and scheduling across possibly many instances.

Treating "make inference faster" as one problem rather than a stack of separable layers is the most common reason performance work stalls — each layer has its own bottlenecks and its own fixes.

From a file on disk to an answer for a user

A trained model is, physically, a large file of numbers. Getting from that file to a working API endpoint that answers a user’s request in a reasonable time involves several genuinely separate layers, and the rest of this roadmap is organised around exactly these layers, one group at a time.

Hardware is the GPU (or GPUs) the model actually runs on, with its own compute throughput and memory bandwidth — covered in the next group. The inference engine is the software that takes a request and the model’s weights and actually computes an output — tokenizing the input, managing the KV cache, running the forward pass, batching multiple requests together — covered in the serving software group. The serving framework sits around one or many engine instances, accepting HTTP requests, routing them to an available instance, and handling what happens when demand exceeds capacity — covered in the production group at the end of this roadmap.

What to watch for

This is the freshest and most complete account found of what an inference server actually is, layer by layer — HTTP input and output, the tokenizer, the scheduler, the GPU model runners underneath — explaining concretely why the scheduler in particular tends to be where the real bottleneck sits. At about an hour it is the longest single video in this roadmap; treat it as a reference to revisit rather than something to watch start to finish in one sitting, and feel free to jump to the chapter that matches whatever you are currently trying to understand.

What actually happens inside the engine layer

Zooming into just the inference engine reveals its own set of distinct steps, each with its own cost. A request’s text first passes through a tokenizer, converting it into the token ids the model actually operates on — cheap, but not free, and a surprisingly common source of subtle bugs when a tokenizer mismatches the model it is paired with. The scheduler then decides which requests get processed in this step and how they are batched together, a decision covered in depth in continuous batching. The model itself executes its forward pass, split into the prefill and decode phases covered in the next group. The scheduler is frequently where real bottlenecks hide, because a slow or naive scheduling decision can leave a GPU underutilised even when every other layer is working efficiently — which is exactly the point the video’s long middle section is built around.

Why the layering matters in practice

The most common mistake in performance work on an inference system is treating “make it faster” as one undifferentiated problem rather than asking which layer is actually the bottleneck right now. A GPU running at its memory-bandwidth limit needs a different fix — covered in memory-bound vs compute-bound — from a request sitting in a queue because the serving framework is routing poorly, covered in routing and load balancing. Diagnosing which layer is actually slow, rather than guessing, is most of the skill this entire roadmap is building toward — and the benchmarking lesson later in this roadmap covers exactly how to find out.

Check yourself

01What is the practical difference between an inference engine and a serving framework?

The inference engine is concerned with running the model itself efficiently on the hardware — batching requests, managing the KV cache, executing the forward pass. The serving framework sits around it, handling incoming HTTP requests, routing them to an available instance, and coordinating multiple engine instances together.

02Why does it help to think of "making inference faster" as a stack of layers rather than one problem?

Each layer has entirely different bottlenecks and entirely different fixes — a GPU memory-bandwidth limit is solved differently from a request-routing inefficiency, which is solved differently from a tokenizer being unnecessarily slow. Treating them as one undifferentiated problem makes it hard to know where to actually look.

Save your progress

Your account

Delete your account?