Vector memory and RAG for agents

From exact lookup to meaning-based lookup

The previous lesson left an open problem: once an agent has stored hundreds or thousands of episodic memories, how does it find the one relevant to the task in front of it right now? A naive answer is keyword search — store memories as text, search for exact matches on words in the current query. This fails constantly in practice, because people rarely phrase the same idea the same way twice. A memory stored as “the client wants the deck delivered before the board meeting” won’t match a later query about “presentation deadline,” even though the two sentences describe the same fact. Vector memory solves this by searching on meaning instead of wording.

The mechanism: embed, search, retrieve

The pipeline has three steps, and each one is doing distinct work.

Embed. Every piece of text an agent wants to remember — an episode, a document chunk, a note — gets passed through an embedding model, which turns it into a fixed-length vector of numbers: a point in a high-dimensional space where texts with similar meaning land close together and texts with different meaning land far apart. This happens once, at write time, and the resulting vector is what actually gets stored, alongside the original text.

Search. When the agent needs to recall something, its current query — “what did the client say about the deadline?” — gets embedded the same way, producing a query vector. A vector database then compares that query vector against every stored vector using a distance measure (cosine similarity is the common choice) and returns the handful of stored vectors closest to it — the nearest neighbors.

Retrieve. Each of those nearest-neighbor vectors points back to the original text it was computed from. That original text — not the vector itself, which is meaningless to the model as a lookup key — is what gets pulled into the agent’s context window for use in generating a response.

This exact three-step pattern — embed, search, retrieve — is also what RAG (retrieval-augmented generation) means when applied to an agent’s own memory rather than to an external document corpus: the “generation” step is the agent’s response, and the “retrieval” step is this vector lookup run just before it.

A worked example

Suppose an agent has stored the episodic memory “the client wants the deck delivered before the board meeting on the 14th” as one vector, among thousands of other stored memories, back when that conversation happened. Weeks later, in an unrelated session, the user asks the agent “when’s that presentation due?” Neither “presentation” nor “due” appears anywhere in the stored memory — a keyword search would return nothing. But the embedding model places “presentation due” and “deck delivered before the board meeting” close together in vector space, because they express closely related meaning. The nearest-neighbor search finds the stored vector, pulls back its original text, and the agent can answer “before the board meeting on the 14th” without the user having to repeat context that was only ever mentioned once, in a different conversation, in different words.

The limitation: similarity has no clock

Vector search measures one thing only — how close two pieces of text are in meaning — and nothing about that measurement knows which of two similar memories is more recent or more trustworthy. Extend the example: suppose three weeks after that conversation, the client emails to move the deadline to the 21st, and that update also gets embedded and stored as a new memory. Both the original “14th” memory and the updated “21st” memory describe near-identical content — the same client, the same deck, the same board meeting — so both sit close together in vector space and both are legitimate nearest-neighbor matches to the same query. Pure similarity search has no mechanism for preferring the newer one; it can just as easily rank the stale “14th” memory first, especially if its wording happens to be a slightly closer match to how the query was phrased. The agent would then confidently tell the user the wrong date, not because retrieval failed, but because retrieval succeeded at the only thing it was built to measure.

This is why production memory systems rarely rely on raw similarity alone. Common fixes include attaching a timestamp to every stored vector and boosting recent matches, filtering out memories explicitly marked as superseded, or re-ranking the top similarity results with a secondary step that checks for contradictions. None of these are part of the core embed-search-retrieve mechanism — they’re guardrails bolted on specifically because similarity search, by itself, cannot tell old from new.