Fine-tuning

What it is for

Fine-tuning continues training a pretrained model on your own examples, nudging its weights toward a specific target behaviour. It is reliably excellent at teaching form — a consistent output structure, a particular house voice, a narrow classification distinction, a tool-use pattern the base model otherwise keeps getting subtly wrong — and reliably poor at installing new facts, for the reasons the rag-vs-fine-tuning lesson covers in depth. The most economically interesting application is not making a model smarter in any general sense, but making a considerably smaller model sufficient for one specific job: a tuned small model that matches a frontier model’s quality on your one narrow task can cut both cost and latency by an order of magnitude, which is a result worth real engineering effort in a way that “teach it our internal documentation” almost never is.

Full fine-tuning, LoRA, and QLoRA

Full fine-tuning updates every single parameter in the model. It is the most expressive option available and demands memory sufficient to hold not just the model’s weights but also the gradients computed during training and the optimiser’s own internal state — together typically several times the model’s base size in memory. For anything beyond a genuinely small model, this requires serious, often multi-GPU hardware. LoRA (low-rank adaptation) instead freezes the original pretrained weights entirely and inserts small additional matrices alongside them that are the only thing actually trained — a tiny fraction of a percent of the total parameter count. Memory requirements collapse correspondingly, and critically, the output of training is a small, separate adapter file rather than an entire new copy of the model, which means one base model deployed once can serve many different tuned behaviours by swapping which small adapter is loaded alongside it at serving time — a pattern particularly valuable for serving many customers, each with their own narrow tuned behaviour, off a single shared base model. QLoRA goes one step further, quantising the frozen base model down to four-bit precision while still training the LoRA adapter itself at higher precision, which brings genuinely useful fine-tuning within reach of a single consumer-grade GPU. For nearly every application-engineering purpose covered in this roadmap, LoRA is the sensible starting point, with full fine-tuning reserved for cases where a measured comparison has actually shown the adapter approach to be the limiting factor.

Alternatives to supervised fine-tuning, briefly

Supervised fine-tuning on example input-output pairs is the most common starting point, but it is not the only training technique in active use. RLHF (reinforcement learning from human feedback) trains a model using comparisons between pairs of its own outputs, rated by human preference, rather than a single fixed correct answer for each input — this is closer to how a model’s broader instruction-following and conversational behaviour is typically refined by model providers themselves than how narrow task-specific fine-tuning is usually done. DPO (direct preference optimisation) achieves a broadly similar outcome to RLHF through a simpler, more directly optimisable training objective computed straight from preference pairs, without needing a separate reward model trained as an intermediate step. These are mentioned here mainly so the terms are recognisable; application-level fine-tuning for a specific narrow task is overwhelmingly supervised fine-tuning on curated input-output examples, not these more involved preference-based training techniques.

The data flywheel, and catastrophic forgetting

Production traces — the real inputs and the corrected or approved outputs your system has already handled — are frequently the best possible source of fine-tuning data, since they reflect the actual distribution of inputs the tuned model will face in practice rather than a hand-curated and inevitably narrower set. This creates a genuine flywheel: production use generates data, that data improves a tuned model, the improved model handles production better, and better handling generates still more useful data. A real risk worth naming directly is catastrophic forgetting — a model trained hard toward one specific narrow behaviour can measurably degrade on other, unrelated tasks it previously handled perfectly well, simply because intensive training toward one objective overwrites some of what the model had previously learned elsewhere. An eval set covering what you did not deliberately intend to change, not only what you did intend to change, is what actually catches this before it reaches production.

Evaluate the baseline before you start

Hold out a genuine test set before any tuning begins, and measure the untouched base model’s performance on it first. This establishes the comparison that actually tells you whether fine-tuning helped at all — a surprising number of fine-tuning efforts never establish this baseline measurement and consequently cannot honestly say what the considerable effort of tuning actually bought them, beyond a vague sense that the outputs look better.

Data quality over data quantity

Quality dominates quantity here, consistently and by a wide margin. A thousand carefully checked, genuinely representative examples routinely outperform fifty thousand examples scraped together with less care, because the model learns the systematic mistakes present in a training set just as precisely and just as reliably as it learns the behaviour you actually intended to teach it. Every consistent error pattern in your training examples becomes a learned, consistent behaviour in the resulting tuned model.