Local and edge inference

A completely different budget

Everything covered so far in this roadmap assumes datacenter-class hardware — GPUs with tens or hundreds of gigabytes of memory, abundant power, and a fast network connection to whoever is calling them. Running inference on a phone, a laptop, or an embedded device operates under a budget that is smaller by orders of magnitude on every one of those dimensions, which changes which techniques from earlier in this roadmap matter most and how aggressively they have to be applied.

What to watch for

A concrete, current walkthrough of running a real model on an iPhone using Apple’s MLX framework, including exactly why aggressive quantization down to four or even fewer bits is necessary at this scale, and where output quality visibly starts to degrade as precision drops further. Watching the actual tradeoff play out on real hardware is more useful here than an abstract description of the constraints.

Why quantization gets pushed further at the edge

Quantization, covered later in this roadmap, reduces a model’s memory footprint by lowering the numeric precision its weights are stored at — see model memory footprint for the arithmetic. A datacenter deployment might comfortably use 16-bit or 8-bit precision, trading a small amount of quality for a meaningful memory saving. An edge device frequently has no choice but to go further — to 4-bit or below — simply to fit a useful model into the memory actually available, accepting a larger quality cost than a datacenter deployment would ever need to tolerate.

What you get that no API can offer

Running on-device is not only a compromise forced by weaker hardware — it buys genuine properties a datacenter API cannot match at any price. There is no network round trip at all, which sets a latency floor far below what any remote call could achieve regardless of how fast the server-side inference is. Data never leaves the device, which matters directly for privacy- sensitive applications and for regulatory requirements that forbid sending certain data anywhere at all. And the device keeps working with no network connection whatsoever, which matters for anything that has to function reliably offline.

The specific pressures that make edge hardware different

Beyond simply having less memory, edge devices add constraints a datacenter GPU never has to consider. Power is finite and often battery-backed, which rules out running hardware anywhere near its peak sustained throughput for long, in a way a datacenter rack with dedicated cooling and a stable power supply never has to worry about. Thermal throttling is a real, immediate effect on a phone or a small embedded board in a way it rarely is in a datacenter — sustained heavy inference can cause the device to slow itself down to avoid overheating, which shows up as inconsistent, degrading performance over the course of a long session rather than the steady throughput a server delivers. And shared resources matter in a way they do not on a dedicated inference server: a phone running inference is also running everything else the user expects the phone to do at the same time, competing for the same limited memory and compute.

Where this leaves hardware strategy

Local and edge inference is not a smaller version of datacenter serving; it is a different point on the same tradeoff space covered throughout this group, pushed to its tightest extreme. The same questions — what matters about the hardware, whether you are memory- or compute-bound, how to size for the workload — all still apply, just against a vastly smaller memory and power budget than anything else in this roadmap assumes. The deeper links cover this pushed even further: running genuinely tiny models on robotics-class hardware, and a panel of practitioners comparing the compression techniques that make any of this feasible at all.