Inference Engineering
Running a trained model efficiently in production, not training one. Thirty-two lessons from latency and throughput through hardware, serving software and the techniques that speed generation up, to running it all at scale.
What is inference engineering?Open vs closed modelsThe inference stackTime to first tokenTokens per second and throughputLatency versus throughputLatency percentiles and end-to-end metricsPrefill and decodeKV cache mechanicsAttention variants for servingMixture of expertsGPU architecture for inferenceArithmetic intensity and the roofline modelMemory-bound vs compute-boundChoosing hardwareLocal and edge inferenceInference enginesKernel fusionBenchmarking inference systemsQuantizationSpeculative decodingContinuous batchingFlashAttention and PagedAttentionPrefix cachingModel parallelismDisaggregated servingEmbedding and multimodal inferenceSpeech inferenceAutoscaling and cold startsRouting and load balancingObservability and costTesting and safe deployment
What inference engineering actually is
- What is inference engineering?Running a trained model efficiently in production — a different job from training one, and a discipline that exists because serving at scale got hard.
- Open vs closed modelsCalling a closed model's API versus self-hosting an open-weight one — and how that choice changes what you actually control.
- The inference stackThe layers between a trained checkpoint and an API that serves it — hardware, engine, serving framework — and where performance work happens at each.
The metrics the whole discipline optimizes
- Time to first tokenHow long before the first output token appears, why prefill dominates it, and why it is the latency users actually feel.
- Tokens per second and throughputPer-request generation speed versus aggregate system throughput — two different numbers that can move in opposite directions.
- Latency versus throughputBatching more requests raises aggregate throughput but can raise individual latency — tuning a serving system means picking a point on that curve.
- Latency percentiles and end-to-end metricsWhy averages hide the real user experience — p50/p95/p99 latency, and measuring the whole request path, not just model compute.
What a serving engine actually computes
- Prefill and decodeThe two phases of generating a response — prefill processes the prompt in parallel, decode generates tokens one at a time — with opposite performance profiles.
- KV cache mechanicsWhat is actually stored in the KV cache, why it exists, and how its size scales with context length, batch size and model size — from the serving engine's side.
- Attention variants for servingMulti-head, multi-query and grouped-query attention — why reducing the number of KV heads became a serving-performance decision, not just a modeling one.
- Mixture of expertsSparse MoE models route each token to a subset of expert sub-networks, changing the compute/memory tradeoff versus a dense model of similar size.
What the chip underneath actually limits
- GPU architecture for inferenceWhat actually matters about a GPU for serving — compute throughput, memory bandwidth, memory capacity — and how hardware generations change these numbers.
- Arithmetic intensity and the roofline modelThe ratio of compute to memory movement an operation needs, and the roofline model that predicts whether it is compute- or memory-bound on given hardware.
- Memory-bound vs compute-boundWhy LLM decode is usually memory-bandwidth-bound rather than compute-bound, and what that implies about where optimization effort actually pays off.
- Choosing hardwareSizing and procuring GPUs for a real workload — matching hardware to model size, batch size and latency targets, not just buying the newest chip.
- Local and edge inferenceRunning inference on consumer hardware or at the edge rather than a datacenter — far tighter memory and power constraints, and the tradeoffs that forces.
The layer between a model and an API
- Inference enginesWhat a dedicated inference engine like vLLM, TensorRT-LLM or SGLang does beyond running the model forward — and why it beats calling a raw framework directly.
- Kernel fusionCombining multiple small GPU operations into one kernel to avoid repeatedly reading and writing memory between them.
- Benchmarking inference systemsMeasuring a serving system's real performance under realistic load — the gap between a synthetic single-request number and actual production traffic.
Techniques that buy real speed without new hardware
- QuantizationReducing numerical precision to cut memory and increase speed — the quality cost, and post-training quantization versus quantization-aware training.
- Speculative decodingA small, fast draft model proposes several tokens at once, verified in parallel by the full model — speeding up generation without changing its output.
- Continuous batchingDynamically adding and removing requests from a running batch as they arrive and finish, keeping GPUs busy under real, bursty traffic.
- FlashAttention and PagedAttentionFlashAttention restructures attention to avoid wasted memory traffic; PagedAttention manages the KV cache like virtual memory to reduce fragmentation.
- Prefix cachingReusing the KV cache computed for a shared prompt prefix across multiple requests, at the serving-engine level.
Past one GPU, and past text generation
- Model parallelismSplitting a model too large for one GPU across several — tensor, pipeline and expert parallelism — and the different tradeoffs each one makes.
- Disaggregated servingSeparating the prefill and decode phases onto different hardware pools, since they have such different performance characteristics.
- Embedding and multimodal inferenceServing embedding models and vision-language models — different bottlenecks than text generation, since there is often no multi-step decode at all.
- Speech inferenceServing speech recognition and text-to-speech models, where real-time conversational latency requirements are especially strict.
Keeping it running under real traffic
- Autoscaling and cold startsScaling GPU capacity with demand, and the real cost of a cold start — how long it takes a new instance to load a large model before it can serve anything.
- Routing and load balancingDirecting a request to the right instance — accounting for which model or adapter is loaded, current queue depth, and which serving phase an instance handles.
- Observability and costMonitoring a production inference system — the metrics that matter, attributing cost per request, and catching degradation before users do.
- Testing and safe deploymentRolling out a new model version or serving configuration without downtime, and catching a regression before it reaches all production traffic.