Choosing a model

Four axes, no winner

Capability, latency, cost and control. Every model is a position on those four, and improving one usually costs you another. A frontier model is the most capable and the slowest and most expensive. A small open-weights model is cheap and fast and will fail on the hard tail of your inputs. There is no dominant choice, only a choice that fits the thing you are building — and most applications are not one workload. A cheap model handles routing, extraction and classification; an expensive one handles the step that actually needs judgement. Picking per call, a pattern sometimes called a model router or cascade, is usually better engineering than picking one model for the whole product: send every request first to a small model, and only escalate to a larger one when a cheap confidence signal — a low score, a flagged uncertainty, a failed validation — says the easy model was not enough.

Benchmarks, and specifically why they mislead

Public leaderboards suffer from problems worth naming individually, not just gesturing at. Contamination means some fraction of a benchmark’s test questions have very likely appeared, verbatim or near enough, in a model’s training data, inflating its apparent performance on exactly that benchmark without telling you anything about genuinely novel inputs. Aggregation means a single headline score averages across wildly different task types, so two models can tie on the overall number while one is far better at your specific task and far worse at everything else the benchmark measured. Crowd-voted arenas, where people compare two anonymous responses and pick a favourite, suffer their own bias: people tend to reward longer, more confident-sounding answers regardless of correctness, which is the same verbosity bias covered in LLM as judge. And narrow synthetic tests like needle-in-a-haystack retrieval checks measure one specific failure mode and say nothing about reasoning, tool use or your actual document format.

The cost arithmetic of self-hosting

Running an open-weights model yourself trades a per-token bill for fixed infrastructure cost, and the trade only pays off at real, steady volume. A model’s parameters occupy GPU memory roughly in proportion to their count and their numeric precision — a seven-billion-parameter model in 16-bit precision needs roughly fourteen gigabytes just to hold the weights, before accounting for the KV cache of whatever conversations are being served concurrently, with quantization to 8-bit or 4-bit cutting that memory requirement roughly in half or further at some cost to quality (see model memory footprint). Below a threshold of sustained request volume, a GPU sitting partly idle costs more per answered request than paying a provider per token; above that threshold, self-hosting wins decisively because the hardware cost is fixed regardless of how hard it is worked. The crossover point depends on model size, hardware pricing and your actual traffic pattern, which is exactly why it needs calculating for your situation rather than assumed from a rule of thumb.

Licensing is part of the decision, not an afterthought

“Open weights” does not mean one license. Some open-weight models permit unrestricted commercial use; others restrict use above a revenue threshold, forbid using the model’s outputs to train a competing model, or carry other conditions that matter the moment the product generates real revenue. This is a question to settle before building a dependency on a model, not after.

Building in the ability to switch

Whatever you choose today will not be what you run in a year — model releases, pricing and your own volume all shift faster than most product roadmaps. An abstraction layer between your application code and the specific provider API, even a thin one, is what turns a model change into a configuration update rather than a rewrite, and an eval set run against every candidate is what answers the choice with evidence rather than a leaderboard position.