Bias and fairness

Where it comes from

A model learns the statistical distribution it was trained on. If that training text over-represents certain populations or viewpoints, encodes historical patterns of who tends to hold which job or occupy which role, or simply contains far more text in one language than another, the model reproduces all of that faithfully — not as some defect in the training process, but as the direct, expected consequence of learning accurately from the data it was shown. A product then adds its own additional layer of decisions on top: which examples were chosen for a few-shot prompt, which documents a retrieval index happens to contain and in what proportion, where a classification threshold was set, who was actually in the room when a default behaviour was decided. A model that is itself close to perfectly neutral, if such a thing existed, can still produce clearly unfair outcomes once deployed carelessly — and most of the decisions genuinely available to an application engineer, who typically does not control the underlying model’s training, live in this second, product-level layer rather than in the model itself.

The categories worth distinguishing

Representation bias occurs when certain groups are simply underrepresented or stereotypically portrayed in training data or in a retrieval corpus, so a system’s output quality or framing differs for a group that was less visible during training. Allocation bias occurs when a system’s outputs materially affect who gets access to some real resource or opportunity — a loan recommendation, a job-candidate screening — and that access differs systematically by group in a way that is not justified by legitimate, relevant differences. Stereotyping occurs when a model’s output reflects or actively reinforces a cultural stereotype even when no allocation decision is directly involved at all — describing a profession with a default assumed gender, for instance. Quality-of-service disparities occur when a system simply performs objectively worse for some groups than others on the exact same underlying task — a speech system that transcribes some accents noticeably less accurately than others is a clear, measurable example of this specific category.

Measurement techniques that actually surface these

Disaggregated evaluation reports accuracy and quality metrics broken out separately by relevant group, rather than as a single aggregate number that can hide a serious failure concentrated in one population behind a perfectly acceptable overall average. Counterfactual testing changes one attribute in an otherwise identical input — swapping a name associated with one demographic group for a name associated with another, while holding everything else about the request completely constant — and checks whether the output changes in ways that should have no legitimate bearing on the response, which is frequently one of the most revealing and surprisingly easy tests to run.

The tension between personalisation and fairness

A system built to personalise its responses to an individual user, by design, differentiates its behaviour based on information about that specific person — which is directly in tension with a fairness goal of treating people the same regardless of group membership. There is no universal resolution to this tension; the right balance depends heavily on what the product is actually for and what the real-world stakes of a difference in treatment are. A recommendation engine personalising content suggestions sits very differently on this question than a system influencing credit or employment decisions, and the acceptable degree of differentiated treatment should be a considered, explicit product decision rather than an unexamined accident of how the system happened to get built.

Mitigation beyond measurement

Several concrete levers exist once a disparity has actually been measured. Explicit prompt-level instructions asking the model to apply a consistent standard regardless of identifying details present in an input have some measurable effect, though they are not a complete fix on their own. Deliberately curating a retrieval corpus for balanced, representative coverage across the perspectives and groups a system’s answers touch on can meaningfully reduce representation bias flowing through to retrieved context. Post-processing output — checking it against a disparity metric before it is returned, and regenerating if that check fails — can catch cases the generation step alone did not avoid. None of these is complete on its own, and in practice they are combined rather than treated as alternatives to pick between.

What you can actually do day to day

Measure outcomes broken out by group, never only in aggregate — an overall accuracy figure reliably hides exactly the concentrated failures that matter most. Examine a RAG system’s retrieval corpus directly for whose voice and perspective it actually represents, since the corpus functions as the product’s effective point of view on any topic it covers. Test explicitly with names and dialects outside whatever the unstated default assumption is — a striking share of real-world failures surface simply by changing a name in an otherwise identical prompt or by rephrasing the same request in non-standard English. Keep a human genuinely in the loop for any decision that materially affects someone’s access to something that matters, rather than fully automating it. And write down what was decided and why, since a fairness choice that is never recorded tends to get silently and inconsistently re-made the next time someone else happens to touch the same threshold or prompt.