GPU architecture for inference

Three numbers, and which one bites depends on the workload

A GPU’s suitability for serving comes down to three characteristics. Compute throughput measures how many floating-point operations it can perform per second — its raw arithmetic speed. Memory bandwidth measures how fast data can move between the GPU’s memory and its compute units — how quickly it can actually feed its arithmetic units with the numbers they need. Memory capacity measures how much data — a model’s weights, plus the KV cache for every concurrent request — it can hold in memory simultaneously. Different workloads are limited by different ones of these three: as covered in memory-bound vs compute-bound, prefill tends to be limited by compute throughput and decode by memory bandwidth, while serving many long conversations at once is often limited simply by memory capacity.

Tensor cores, and why precision matters

Modern GPUs include dedicated tensor cores — hardware specifically built to accelerate the matrix multiplication that dominates both training and inference, at reduced numerical precision (16-bit, 8-bit, or lower rather than the 32-bit precision ordinary arithmetic might use). Running at lower precision trades a small amount of numerical accuracy for a substantial speed gain, which is why quantization, covered later in this roadmap, is as effective as it is — it deliberately targets exactly the precision these specialised cores are fastest at.

What to watch for

Modal’s Charles Frye walks through exactly what matters about a GPU for serving — bandwidth versus compute throughput, tensor cores, low-precision matrix multiplication — using a current hardware generation as the concrete reference point throughout, which grounds the abstract numbers in something real rather than leaving them purely theoretical.

Why hardware generations matter to this roadmap

Each new GPU generation typically improves memory bandwidth and low-precision compute throughput together, and that specific combination is exactly what benefits LLM inference the most, given how often decode is the memory-bandwidth-bound phase. This is why the specific hardware generation a team is serving on genuinely changes the right answers to questions covered elsewhere in this roadmap — what batch size is sustainable, what quantization level is worth using, how many concurrent users one GPU can serve — and why any specific numbers quoted in this field age quickly as new generations arrive. The concepts in this lesson outlast any specific chip; the exact numbers do not.

Why these three numbers rarely improve at the same rate

It is tempting to think of a GPU generation as uniformly “better” than the last, but compute throughput, memory bandwidth and memory capacity typically improve at different rates from one generation to the next, and not always in proportion to each other. A generation can deliver a large jump in raw compute while improving memory bandwidth only modestly, which actually narrows the advantage for a memory-bound workload even as the chip’s headline specifications look dramatically better. This is exactly why reading a GPU’s marketing specifications in isolation can mislead: the number that matters for your specific workload might be the one that barely moved, while the one that moved the most is largely irrelevant to what you are actually running.

Where this feeds into the rest of the roadmap

Understanding these three characteristics is the foundation for the roofline model in the next lesson, which formalises exactly how compute throughput and memory bandwidth together predict whether a given operation will run at its compute limit or its memory limit — and for choosing hardware later in this group, which is this lesson’s three numbers applied to an actual sizing decision.