Inference engines

Everything a naive forward pass does not do for you

Calling a model framework directly — load the weights, run a forward pass, read the output — technically produces inference, but it does none of the work that makes serving real traffic efficient. It processes one request at a time with no batching. It recomputes or wastes memory on the KV cache with no particular care. It runs through whatever kernels the framework ships with, which were generally written with training, not serving, in mind. A dedicated inference engine exists specifically to do all of this well: vLLM, TensorRT-LLM, and SGLang are the three most widely used in production today, and each automates continuous batching, KV cache memory management, and serving-optimized kernel execution as its core job.

What to watch for

An SGLang core maintainer and a Baseten engineer explain SGLang’s scheduling, caching and kernel-level design choices directly from the project team, on the official conference channel — a genuine inside view of how one of these three engines is actually built, which is more valuable here than a surface-level feature comparison. It runs long, with a codebase tour and audience questions, so treat it as something to dip into at a relevant timestamp rather than watch in full. vLLM and TensorRT-LLM, the other two engines this lesson names, are covered by the deeper links.

Why the gap between engines matters less than it sounds

These three engines differ in real ways — their scheduling policies, which hardware vendors they are most optimized for, specific advanced features one has that another lacks — and reasonable engineering teams choose between them based on real tradeoffs covered partly in benchmarking inference systems. But the gap between any of them and a naive, unoptimized implementation is typically far larger than the gap between the engines themselves. The first and most important decision is using a dedicated engine at all; which specific one is a real but secondary question.

Formats: what actually moves between systems

A trained model is distributed as a checkpoint in one of several formats. Safetensors has become the default on the Hugging Face ecosystem specifically because it stores only raw tensor weights with no embedded executable code, which makes it fast to load and safe to download from untrusted sources — unlike Python’s older pickle format, which can execute arbitrary code on load. ONNX is a portable interchange format that captures not just weights but the model’s full computation graph, intended to let a model trained in one framework run in a different runtime altogether. Framework-native formats (a raw PyTorch checkpoint, for instance) are the simplest to produce but the least portable across tooling.

Converting between these is a real engineering step, not a trivial file copy: a format encodes architecture metadata and tokenizer configuration alongside the raw numbers, and an incomplete or careless conversion can silently drop or misinterpret some of it, producing a model that loads without any error at all and still behaves subtly differently from the original. Whichever inference engine you choose typically has a preferred format, and picking a model distribution that already matches it avoids an entire category of conversion bugs before they happen.