Privacy and data handling

Count the copies

A single request makes more copies of user data than people expect. The text goes into a prompt. The prompt travels to a provider, who may log it depending on their policy. A trace store keeps the entire exchange, verbatim, for debugging purposes. A cache may hold the stable prefix. An eval set may later be built directly from exactly these same production traces. A debugging session puts the raw content on an engineer’s screen while investigating an unrelated issue. Each of these copies has its own lifetime, its own access control, and its own answer — or lack of one — to “what happens to this when the user asks us to delete their data.” Most real privacy problems in LLM systems are not dramatic breaches; they are simply a copy nobody accounted for when the system was first designed.

Data minimisation, as a set of concrete techniques

The strongest control available is simply not transmitting sensitive data in the first place. Field-level redaction strips specific fields the current task does not actually need before anything is sent to a model — a support summarisation task rarely needs a customer’s full address, for instance, even if the source ticket contains it. Tokenisation or pseudonymisation replaces a real identifier with an opaque reference the model can reason about structurally — “customer 4471 opened a ticket” — while the mapping back to the real identity is kept entirely on your own side and never sent to the model at all. Where feasible, on-device or local processing for the most sensitive categories of data avoids transmitting it to any external API whatsoever, at the cost of generally needing a smaller, locally-hosted model rather than a frontier one.

Redaction is reduction, not anonymisation

Removing obvious identifiers like names and email addresses genuinely helps and should be done as standard practice, but it is not the same thing as making data anonymous, and treating it as equivalent is a common and consequential mistake. A postcode, a date of birth and a job title, none of which looks individually sensitive, together identify a striking majority of people in many populations when combined — a well-known finding from privacy research on exactly this kind of quasi-identifier combination. Genuine anonymisation, where it is actually required, needs considerably more care than stripping a handful of obviously named fields, and differential privacy — a formal mathematical framework for adding calibrated noise so that no individual record can be confidently inferred from an aggregate result — is the serious research answer to this problem where it is actually needed, though it remains uncommon in everyday LLM application engineering today.

Mapping data-subject rights to system components

Where privacy regulation grants individuals rights over their own data — commonly the right to access what is held about them, the right to have it deleted, and the right to receive a portable copy of it — each of these rights has to be mapped concretely to every single system component identified in the “count the copies” section above, not only to the primary database. A deletion request that successfully removes a user’s row from the main application database but leaves their data intact in a trace store, a cache, or an eval set built from old production traces has not actually fulfilled the deletion in any meaningful sense, even though the most visible, obvious copy was correctly handled.

Provider terms, read rather than assumed

Read the actual terms governing your specific deployment rather than assuming based on a provider’s general reputation or a blog post, and do this before launch rather than discovering the answer under pressure later. The load-bearing questions: is this data used to train or improve the provider’s own models; how long is it retained, and by whom can it be accessed during that retention window; in which jurisdiction is it physically processed and stored; and does an enterprise agreement, where one exists, change any of these answers relative to the provider’s default consumer or standard API terms. Consumer-facing and API-level terms from the same company are frequently different for the same underlying model, so the relevant answer is specifically the one governing your actual deployment, not whatever a general search happens to surface. A number of providers now offer an explicit zero-data-retention option for enterprise customers, under which request content is not persisted at all beyond what is strictly needed to serve that one response — worth asking about directly wherever the sensitivity of the data involved justifies it.

Enforcing permissions where retrieval happens

In a RAG system specifically, permissions have to be enforced during the search itself, filtering by the requesting user’s actual access rights before candidates are even returned, not afterward by hoping the generation step simply declines to use something it was not supposed to see. An index capable of returning a document the current user is not authorised to see has already leaked that document’s content into the context the moment retrieval completes, regardless of anything the prompt subsequently instructs the model to do or not do with it.

What this lesson deliberately does not cover

Regional data-residency requirements and the broader contractual terms appropriate in a data-processing agreement with a vendor are genuinely important, are highly specific to the jurisdictions and industries involved, and change often enough that anything written here in general terms would risk actively misleading a reader relying on it. Where these apply to your situation, they are properly a question for whoever at your organisation answers legal and compliance questions, and are worth raising early in a project rather than discovered as a blocker immediately before launch.