AI Memory Layer: Designing Persistent Memory for Agents

What should an agent remember after a conversation ends? Saving everything is easy to imagine. Deciding what should influence the next answer takes more care.

An AI memory layer is the part of an application that selects, stores and retrieves information for an agent to reuse beyond the current interaction. It also needs a way to correct or remove what it keeps. Persistent memory survives a restart, but persistence alone doesn’t make a memory useful.

In Key Concepts for Implementing Memory in AI Agents, I covered the main concepts. Here, let’s follow one piece of feedback from a conversation to its possible use later.

What is worth remembering?

Consider a hypothetical example. Someone working on a document tells an assistant:

For this email, use a more direct tone because the audience already knows the company.

There are three useful parts: the direction, the scope and the reason. Dropping any of them changes what the person asked for.

“Use a direct tone” is too broad. “The company prefers direct communication” invents a company-wide rule. The original comment concerns one email and an audience that needs less introduction.

Memory is a selective interpretation of what happened, not a second transcript. A temporary instruction can be worth retaining for later revisions of that email without becoming a permanent preference. Ask: what future task would this help with, and under what conditions?

What exactly do we keep?

A memory note could look like this. It is an example, not a database schema:

• Feedback: use a more direct tone in this email.
• Applies to: revisions of the email being discussed.
• Reason given: the audience already knows the company.
• Human source: “For this email, use a more direct tone because the audience already knows the company.”
• Source location: the person’s message in the relevant conversation.

The interpretation makes the note useful to retrieve. The exact supporting excerpt makes it possible to check whether that interpretation is fair. Keeping a route back to the source also helps when “this email” needs clarification.

We also need to distinguish feedback, a proposal and an explicit decision. “Could we make it more direct?” is not the same as “Use this version.” An extractor can misclassify either. Finding an exact quote proves that those words occurred; it does not prove that the extracted meaning is correct.

The assistant’s own suggestion is not human approval. If it proposes a tone change and nobody accepts it, that suggestion should not quietly become a remembered decision.

When do we capture it?

We can capture a clear instruction during the interaction or extract candidates afterward. The choice depends on how soon the information needs to be available.

Background extraction lets the reply continue without waiting for every memory operation. It also creates two separate kinds of progress: the assistant may have answered while memory processing is still pending. A failed extraction should not look like a successful pass that found nothing worth keeping.

Make those outcomes observable so they can be inspected or retried. Otherwise, a user may believe a correction was saved when only the reply succeeded. Extraction should produce candidates that the application can validate, not silently turn every sentence into a lasting rule.

How does the next interaction use it?

Suppose the person returns to revise that email. Start by finding notes applicable to that document and checking current access. A few scoped records may be enough; vector search is not a prerequisite for persistent agent memory.

Three steps are easy to confuse:

Stored information → retrieved candidates → current model context

A saved note might not be retrieved. A retrieved note might be excluded from the prompt because it is irrelevant, disputed or too large. The model can use only the material actually supplied to it in that invocation.

Conversation history records the exchange. Context is the input supplied for a particular model call. Retrieval-augmented generation, or RAG, brings relevant material into that input; it can retrieve documents or memory notes, but does not decide what deserves to be remembered.

For a multi-step run, one design choice is to freeze the selected memories at the start. That gives its steps a consistent view while background extraction continues. Another task may need an explicit refresh. Neither choice is universal; the important thing is knowing which view an answer used.

If retrieval fails or returns only part of the available material, don’t present that as complete recall. “I found no relevant notes” and “I couldn’t check the notes” mean different things.

Permissions belong at both read and write time. A remembered statement that someone manages a document is not authority to access or change it. The application must check current permissions independently.

What if people disagree?

Now imagine another person asks for a more introductory version. Is that a contradiction, a different audience or another email?

Keep enough scope and source information to tell the difference. If both instructions concern the same revision and conflict, surface the disagreement rather than assuming the newest message wins. The assistant can ask which direction applies instead of blending incompatible instructions.

An explicit correction may replace an earlier note under the application’s policy. That still requires deciding who can make the correction and what it applies to. Updating a note should leave an understandable path back to the change; obsolete notes also need a way to be retired or deleted.

Where does a graph help?

A graph can connect a memory to its source message, document, author and related decisions. That helps answer questions such as “Which feedback belongs to this document?” or “What supports this decision?”

Those relationships do not make a claim true. A graph does not supply reasoning or semantic ranking by itself. It organizes connections; the application still needs rules for access, relevance and conflicting interpretations.

Start with simple records if they answer your questions. Add relationships when following them becomes useful. My experimental Markdown knowledge graph pipeline explores document relationships, not a complete agent memory system.

A few practical checks

Before choosing more infrastructure, try a small set of conversations:

• Does feedback about one email stay limited to that email?
• Can someone inspect the human words behind a note?
• Does an unaccepted assistant proposal remain a proposal?
• Are extraction failure and an empty result distinguishable?
• Does revoked access prevent both retrieval and writes?
• Does conflicting feedback prompt clarification rather than a silent overwrite?

The useful question is not how much the agent can store. It is whether the right information reaches the next task with its meaning and limits intact.

For further reading: LangChain’s memory concepts, LangGraph’s persistence guide and the retrieval documentation.