AI Agent Memory: Short-Term, Long-Term and Retrieval Measured

This post is for engineers adding memory to an agent that talks to the same user for weeks. AI agent memory looks like a storage problem at first: keep the conversation, search it later. However, the hard part is time. Users change their minds, move regions, upgrade databases and ask you to forget their phone number. So instead of another taxonomy, this post replays one 30-session relationship through ten memory policies. It then checks whether each one returned the current fact, a superseded one, both, or something the user asked to delete. The results favor a design that most tutorials skip, and the reason is cheap to fix.

The numbers below came from this environment:

Measured 2026-11-06
Windows 11 Pro 25H2 (build 26200), AMD Ryzen 5 8600G (6C/12T), 16 GB RAM
Dart SDK 3.9.0 (stable), package:path 1.9.1
Tokenizer: o200k_base vocabulary (SHA-256 446a9538...cfb1a2d), the Dart port
  from the context engineering bench
Scenario: 30 sessions, 360 exchanges, filler replies quote Dart SDK 3.9.0 lib/ sources, seed 42

No model was called. The bench measures what each memory policy puts in front of the model, which is deterministic, and leaves what a model does with it out of scope. Two consecutive runs produced byte-identical output, and one run takes about 13 seconds.

What Is AI Agent Memory?

AI agent memory is everything an agent can recall beyond the current model call. Short-term memory is the running conversation of one session. Long-term memory is information saved across sessions, such as user preferences and project facts. Retrieval is the mechanism that decides which saved items return to the context window for a given question.

The three layers answer different questions. Short-term memory answers “what did we say five minutes ago”. Long-term memory answers “what do I know about this user”. Retrieval answers “which of those things matters right now”. Most frameworks name these layers explicitly. For instance, the LangGraph persistence docs describe checkpointers as “short-term, thread-scoped memory” and stores as “long-term, cross-thread memory, including user preferences, facts, and shared knowledge.”

The site’s earlier post on context engineering measured what happens inside one long session. This one picks up where that left off: what survives once the session ends, and whether it is still true when it comes back.

Short-Term vs Long-Term Agent Memory at a Glance

	Short-term memory	Long-term memory	Retrieval
Scope	One session or thread	Across sessions	Per query
Typical store	Message list, checkpoint	Profile file, key-value store, fact log	BM25 index, vector index
Who writes it	The loop, automatically	An extractor or the model via a tool	Nothing, it reads
Main failure	Forgets at session end	Keeps facts that stopped being true	Misses reworded questions
Example	LangGraph checkpointer	LangGraph store, Claude memory tool	pgvector, Elasticsearch

The Bench: 30 Sessions, 15 Facts, Ten Memory Policies

The scenario is one developer working with a coding assistant over 30 sessions, one per working day, with 12 exchanges per session. Most exchanges are ordinary questions, such as how to profile a slow query or why a test is flaky. In each of those, the assistant replies with 30 lines of real Dart SDK source, so the filler has realistic token weight and vocabulary.

Fifteen facts are planted in user messages along the way. Eight of them change later, and two are personal details the user asks the assistant to forget:

Fact	Stated	Changed	Example values
Deploy region	day 1	day 14	eu-west-2, then eu-central-1
Postgres version	day 2	day 19	Postgres 15, then Postgres 17
Test runner	day 3	day 11	Jest, then Vitest
On-call engineer	day 6	days 16 and 27	Priya, Marco, Priya again
Node version	day 8	day 22	Node 20, then Node 24
Release branch	day 10	day 24	release/2026.3, then release/2026.4
Time zone	day 13	day 25	Lisbon, then Berlin
Checkout flag	day 15	day 23	false, then true in production
Phone number	day 9	deleted day 20	a +44 test number
Personal email	day 17	deleted day 26	k.dev@example.com

Five more facts never change, such as “always tabs, not spaces” and “use the Money class, never a double”. The distractor questions deliberately share vocabulary with the facts. For example, “Why does the deploy pipeline take so long?” competes with the deploy region, and “How do I set up phone verification?” competes with the phone number.

How Each Probe Is Scored

At the end of days 10, 20 and 30, the bench asks about every fact stated so far. That makes 40 probes per policy, 37 about live facts and 3 about deleted ones. Each probe comes in two forms. The literal form shares words with the original statement, as in “Which region do we deploy to?”. The paraphrase shares few or none, as in “Where does prod live these days?”.

For each probe the bench builds the memory a policy would inject, then classifies it:

// bench.dart (excerpt): one probe, one outcome.
String outcome(Fact f, int s, String ctx) {
  final live = f.versions.where((v) => v.session <= s).toList();
  if (f.deletedAt != null && f.deletedAt! <= s) {
    return live.any((v) => has(ctx, v.value)) ? 'leaked' : 'clean';
  }
  final cur = live.last.value;
  final old = live.reversed.skip(1).map((v) => v.value).where((v) => !cur.contains(v));
  final c = has(ctx, cur), o = old.any((v) => has(ctx, v));
  if (c && o) return 'conflict'; // newest and a superseded value both present
  if (c) return 'current';
  if (o) return 'stale';         // only the superseded value made it in
  return 'missing';
}

has matches the value as a whole token sequence, so “Node 20” does not match “Node 204”. A conflict is not automatically a wrong answer, because every stored line carries its day number and a model could pick the newest. Still, it hands the model a contradiction to resolve, and whether it resolves it correctly is model behavior this bench does not measure.

The Ten Memory Policies

Session buffer only: the current session’s messages. Pure short-term memory.
Full transcript: every message from every session.
Last 20 exchanges: a sliding window that crosses session boundaries.
Fact log, append-only: every user statement that carries a fact, appended as one line, all injected.
Profile, upsert and delete: one line per fact key, overwritten on change and removed on a delete request, all injected.
BM25 top-5 over the transcript: each exchange indexed as one chunk.
BM25 top-5 over the fact log.
Profile plus BM25 top-3 over the transcript: the common hybrid.
The same hybrid, with deleted facts redacted from the transcript.
Profile plus BM25 top-3 over exchanges the profile did not extract: facts live in exactly one place.

Policies 4, 5 and 10 assume a perfect extractor, meaning every fact statement is recognized and, for the profile, mapped to the right key. That assumption flatters them, and the section on extraction below returns to it.

AI Agent Memory Results: Recall, Conflicts and Leaks

Here is the full run:

dart run bench.dart o200k_base.tiktoken "$DART_SDK/lib"

scenario: 30 sessions, 360 exchanges, 15 facts, 8 updated, 2 deleted on request
transcript at day 30: 86058 tokens

policy                                      probe        current conflict stale missing | leaked clean | mean injected  stored@30
session buffer only                         literal            1        0     0      36 |      0     3 |          2863       3062
session buffer only                         paraphrase         1        0     0      36 |      0     3 |          2863  
full transcript, every session              literal           25       12     0       0 |      3     0 |         60731      86058
full transcript, every session              paraphrase        25       12     0       0 |      3     0 |         60731  
last 20 exchanges across sessions           literal            2        0     0      35 |      0     3 |          4809      86058
last 20 exchanges across sessions           paraphrase         2        0     0      35 |      0     3 |          4809  
fact log, append-only, all injected         literal           25       12     0       0 |      3     0 |           323        423
fact log, append-only, all injected         paraphrase        25       12     0       0 |      3     0 |           323  
profile, upsert + delete, all injected      literal           37        0     0       0 |      0     3 |           206        214
profile, upsert + delete, all injected      paraphrase        37        0     0       0 |      0     3 |           206  
BM25 top-5 over transcript                  literal           25       11     1       0 |      3     0 |           840      86058
BM25 top-5 over transcript                  paraphrase         4        1     0      32 |      0     3 |           956  
BM25 top-5 over fact log                    literal           25       12     0       0 |      3     0 |            33        423
BM25 top-5 over fact log                    paraphrase         4        1     0      32 |      0     3 |             9  
profile + BM25 top-3 over transcript        literal           25       12     0       0 |      3     0 |           599      86272
profile + BM25 top-3 over transcript        paraphrase        36        1     0       0 |      0     3 |           747  
profile + BM25 top-3, deletes redacted      literal           25       12     0       0 |      0     3 |           627      86145
profile + BM25 top-3, deletes redacted      paraphrase        36        1     0       0 |      0     3 |           747  
profile + BM25 top-3, facts not indexed     literal           37        0     0       0 |      0     3 |           896      85541
profile + BM25 top-3, facts not indexed     paraphrase        37        0     0       0 |      0     3 |           857

The same results as a comparison table, combining both probe forms (74 live probes and 6 deletion probes per policy):

Policy	Current only	Conflict	Stale or missing	Deleted data leaked	Mean tokens injected
Session buffer only	2	0	72	0 of 6	2,863
Full transcript	50	24	0	6 of 6	60,731
Last 20 exchanges	4	0	70	0 of 6	4,809
Fact log, append-only	50	24	0	6 of 6	323
Profile, upsert and delete	74	0	0	0 of 6	206
BM25 over transcript	29	12	33	3 of 6	840 / 956
BM25 over fact log	29	13	32	3 of 6	33 / 9
Profile + BM25 over transcript	61	13	0	3 of 6	599 / 747
Same, deletes redacted	61	13	0	0 of 6	627 / 747
Profile + BM25, facts not indexed	74	0	0	0 of 6	896 / 857

Where two token figures appear, the first is the literal probes and the second the paraphrases. Five findings follow, and the third and fourth matter most for anyone building a memory layer today.

Short-Term Memory Alone Forgot 36 of 37 Facts

The session buffer recalled one live fact per probe form: the release branch, asked on the same day it was stated. Everything else had been said in an earlier session. The sliding window did barely better at 2 of 37, because 20 exchanges cover less than two days of this scenario. Neither result is surprising. Still, it puts a number on a common shortcut: a chat app that persists the thread and calls it memory remembers roughly the last day and nothing else.

Short-term memory is still essential, because it holds the working state of the current task. It just cannot be the whole design once users come back.

Keeping Everything Kept the Old Answers Too

The full transcript never missed a fact, which is its appeal. However, it produced a conflict on all 12 probes where a fact had changed, and leaked the deleted phone number and email on all 3 deletion probes. At 60,731 tokens per probe on average, it was also the most expensive policy, at nearly 300 times the cost of the profile.

The append-only fact log produced exactly the same recall, conflicts and leaks for 323 tokens. That makes sense, because it is the transcript with the filler removed. In practice, every superseded value and every deleted detail is still in there, sitting next to its replacement. The log even contains the deletion request itself, “Please forget my phone number”, one line below the phone number.

This is not a strawman design. Mem0’s current documentation describes ADD-only extraction: “New memories are added without overwriting or deleting existing memories.” Its evaluation page states the trade-off openly, noting that “older facts are preserved rather than overwritten, so semantically similar prior facts can still surface alongside newer ones.” Append-only buys a full audit trail and simple writes. In exchange, conflict resolution moves to read time, where the model has to do it on every call.

Lexical Retrieval Answered 5 of 37 Reworded Questions

BM25 over the transcript looked fine on literal probes: 36 of 37 probes found the current value, although 11 of those also returned the old one. On paraphrased probes it fell to 5 of 37. The other 32 retrieved code chunks and distractor exchanges that happened to share a word with the question. Over the fact log, paraphrases did worse in a quieter way. Most queries shared no terms with any fact line, retrieved nothing, and injected an average of 9 tokens.

Users rarely phrase a question the way they phrased the fact weeks earlier. “Where does prod live these days?” shares no indexed term with “We moved production to eu-central-1”, because “prod” and “production” are different tokens to a lexical index. This gap is the textbook argument for embeddings, which the site covers in comparing OpenAI, Voyage and Cohere embeddings, and for combining both in hybrid search with BM25 and vectors. Dense retrieval was not measured here, so this bench cannot say how much of the gap it closes.

The One Stale Hit: Retrieval Ranked the Old Fact Above the New One

One literal probe returned only the superseded value. On day 30, “Which Node version do services run on?” retrieved this top five:

BM25 top-5 over transcript, day 30, "Which Node version do services run on?"
  [day 8] user: Services run on Node 20. / assistant: Noted: Node 20.
  [day 3] user: How do I write a migration that adds a column safely? / assistant: Here is the relevant code:
  [day 30] user: How do I write a migration that adds a column safely? / assistant: Here is the relevant code:
  [day 24] user: How do I paginate a large result set? / assistant: Here is the relevant code:
  [day 5] user: Why is the production build bigger than the dev build? / assistant: Here is the relevant code:

The day 8 statement, “Services run on Node 20”, matched “services”, “run” and “node”. By contrast, the day 22 update, “Every service runs on Node 24 now”, matched only “node”, because the index does no stemming. Four code chunks containing “version” and “run” outranked it. Notably, retrieval scores relevance, not recency. Nothing in a similarity score knows that one statement replaced another, so the older phrasing can win whenever it happens to match the question better.

The Common Hybrid Brought the Conflicts Back

The profile alone was the only single-store policy with a clean sheet: 74 of 74 current, no conflicts, no leaks, at 206 tokens. Adding BM25 over the full transcript, the usual way to cover questions the profile does not answer, undid that on literal probes. All 12 literal probes on changed facts came back as conflicts, because retrieval pulled the original statement out of the transcript and placed it next to the profile’s current line.

Redacting deleted facts from the transcript stopped the leaks but left all 13 conflicts. The fix that removed both was structural. Policy 10 indexes only the exchanges the extractor did not turn into profile entries, so every fact lives in exactly one store, the one that handles updates and deletes. It scored 74 of 74 with no leaks, at 857 to 896 tokens per probe, the price of three retrieved exchanges.

Where the Profile’s Perfect Score Comes From

The profile scored perfectly because the bench handed it a perfect extractor. Every fact statement arrived tagged with the right key, so “We moved production to eu-central-1” overwrote the deploy region line rather than creating a second one. Here is the whole store:

// bench.dart (excerpt): one line per key, overwritten on change, removed on delete.
List<String> profile(int s) {
  final byKey = <String, String>{};
  for (final e in upTo(s)) {
    final key = e.factKey;
    if (key == null) continue;
    final f = facts.firstWhere((f) => f.key == key);
    if (f.deletedAt == e.user.session && e.user.text == f.deleteRequest) {
      byKey.remove(key);
    } else {
      byKey[key] = '[day ${e.user.session}] ${e.user.text}';
    }
  }
  return byKey.values.toList();
}

In a real system, deciding that a sentence updates an existing fact, rather than adding a new one, is a model call. It can fail in both directions. If it misses an update, the profile holds two deploy regions and behaves like the append-only log. On the other hand, if it merges two different facts, the profile silently loses one. This bench measures the cost of each storage shape once the extraction is right. It does not measure how often extraction is right, and that number will decide the profile’s real-world score.

So the honest reading is this. Upsert-and-delete storage removes conflicts and leaks by construction, and it is the cheapest policy measured. Append-only storage cannot remove them at all, no matter how good the extractor is.

How to Design AI Agent Memory

Based on these measurements, a practical order of work looks like this:

Keep short-term memory as the thread itself, and do not count on it beyond the current session.
Store durable facts in a keyed profile that supports update and delete, not in an append-only log.
Give every stored line a date, so any conflict that does slip through can be resolved by recency.
Index only what the profile does not hold for retrieval, so a fact never lives in two stores with two lifecycles.
Apply delete requests to every store, including transcripts, retrieval indexes and backups.
Test retrieval with reworded questions, not only with the phrasing the fact was stored in.
Measure extraction accuracy on your own traffic before trusting any of the above.

Step 4 is the one tutorials skip most often. It also cost the least to get right in this bench, and nothing else removed the conflicts.

Implementing the Profile with Claude’s Memory Tool

Anthropic’s memory tool maps well onto the profile design. You declare it as {"type": "memory_20250818", "name": "memory"}, and the model then issues file commands such as view, create, str_replace, insert, delete and rename against a /memories directory. Crucially, str_replace is an upsert and delete is a delete, which is exactly the shape that scored 74 of 74 above.

The documentation is explicit that the tool “operates client-side: Claude requests file operations, and your application runs them.” Consequently, your code owns the storage, and it owns the security. The docs warn that a path such as /memories/../../secrets.env can escape the directory, and that “your implementation must validate every path in every command.” Here is a validated resolver in Dart:

import 'dart:io';

import 'package:path/path.dart' as p;

/// Maps a memory tool path such as `/memories/profile.md` onto [root] on disk,
/// rejecting anything that would land outside it.
String resolveMemoryPath(String root, String requested) {
  if (requested != '/memories' && !requested.startsWith('/memories/')) {
    throw ArgumentError.value(requested, 'path', 'must start with /memories');
  }
  final relative = requested.substring('/memories'.length).replaceFirst(RegExp(r'^/+'), '');
  final base = p.canonicalize(root);
  final resolved = p.canonicalize(p.join(base, relative));
  if (resolved != base && !p.isWithin(base, resolved)) {
    throw ArgumentError.value(requested, 'path', 'escapes the memory directory');
  }
  return resolved;
}

void main() {
  final root = Directory.systemTemp.createTempSync('memories').path;
  for (final path in [
    '/memories/profile.md',
    '/memories/projects/checkout.md',
    '/memories/../../secrets.env',
    '/memories/projects/../../outside.md',
    '/etc/passwd',
  ]) {
    try {
      print('OK      $path -> ${p.relative(resolveMemoryPath(root, path), from: root)}');
    } on ArgumentError catch (e) {
      print('REJECT  $path (${e.message})');
    }
  }
}

dart run memory_paths.dart

OK      /memories/profile.md -> profile.md
OK      /memories/projects/checkout.md -> projects\checkout.md
REJECT  /memories/../../secrets.env (escapes the memory directory)
REJECT  /memories/projects/../../outside.md (escapes the memory directory)
REJECT  /etc/passwd (must start with /memories)

p.canonicalize collapses the .. segments before the containment check, which is why the check happens after resolution rather than on the raw string. Checking the raw string for .. misses encoded and nested variants. In contrast, checking the resolved path catches anything that lands outside the root, however it got there.

The memory tool does not enforce step 4 for you. If the same deployment also indexes full transcripts for retrieval, the conflicts above come straight back. The docs also recommend periodically deleting memory files that have not been accessed in a long time, which limits how long a stale fact can sit unnoticed.

Mapping the Layers onto LangGraph

LangGraph splits the same layers across two components. The checkpointer persists one thread’s state, which is the session buffer in this bench. The store persists items across threads, under namespaces your code chooses. Whether that store behaves like policy 4 or policy 5 depends entirely on how your code writes to it. Writing under a stable key per fact gives upsert semantics, while writing a new item for every extracted statement gives the append-only log and its conflicts.

The LangGraph stateful agents post covers checkpointing in depth. For a durable, queryable store, pgvector in Postgres can hold the profile rows and the retrieval index in one database, which makes step 5’s delete-everywhere rule a single transaction.

Real-World Scenario: The Assistant That Remembered the Old Region

Consider a small platform team that gives its internal coding assistant long-term memory over a few months. The implementation is the common one: every user message goes into a vector index, and the top few matches are injected on each call. It works well at first. The assistant remembers conventions, ticket prefixes and who owns which service, and people stop repeating themselves.

Then the team migrates production to a new region. An engineer mentions it in passing, and for a while the assistant uses the new region correctly, because recent messages rank well. Several weeks later, someone asks for a deploy command phrased much like an early message from onboarding. Retrieval ranks that old message first, as the Node probe did above. The assistant generates a command for the old region. Nothing errors, because the old region still exists, and the mistake surfaces only in review.

The fix in this situation is the one the bench points to. Facts move into a keyed profile with update semantics, the raw messages that stated them leave the retrieval index, and every profile line carries a date. The trade-off is a new moving part: an extraction step that must recognize updates, and that the team now needs to evaluate on its own messages.

When to Use Long-Term Agent Memory

Users return across sessions and expect the agent to know stable preferences and project facts
Facts change slowly enough that an update path can keep up, such as regions, versions and owners
You can store a date with every memory and expose it to the model
There is a clear owner for deletion, including retrieval indexes and backups
The cost of repeating context every session is higher than the cost of an extraction call

When NOT to Use Long-Term Agent Memory

Single-session tasks, where the thread already holds everything the agent needs
Regulated data that users cannot easily inspect or delete from every store
Facts that change faster than the extractor runs, such as live metrics or prices, which belong in a tool call
Teams without a way to evaluate extraction, because silent merges and missed updates go unnoticed
Shared agents where one user’s memories could surface for another without strict namespacing

Common Mistakes with AI Agent Memory

Treating a persisted chat thread as long-term memory, which in this bench recalled 1 of 37 facts
Storing facts append-only and expecting the model to sort out every conflict at read time
Indexing raw transcripts for retrieval next to a profile, which brought back all 12 literal conflicts
Honoring delete requests in the profile but not in the transcript, the retrieval index or backups
Testing retrieval only with the original phrasing, when BM25 found the current value for only 5 of 37 reworded questions
Dropping timestamps from stored memories, which removes the only signal for resolving a conflict
Injecting the full transcript because it never misses, at 60,731 tokens per call in this scenario

For the agent loop these memories feed into, see the guide to building AI agents with tools, planning and execution.

What This Measurement Does Not Cover

No model ran in this bench. It measures which facts reach the context window, not whether a model uses them, so a conflict here may or may not produce a wrong answer. Extraction was perfect by construction, which is the largest gap between these numbers and a production system. Dense retrieval was not measured either, so the paraphrase results describe BM25 only.

The scenario is also one shape. It has one user, 15 facts and short user messages, so a larger profile would cost more than 206 tokens to inject on every call, and a profile with thousands of keys would need retrieval of its own. Token counts use one tokenizer, and the leak check only looks for exact values, not paraphrased personal details. The bench reruns in about 13 seconds, and it will be re-run on the next major change to the memory tools or extraction behavior it cites, or at 12 months.

Conclusion

AI agent memory breaks on change, not on storage. Across 30 sessions, append-only memory and full transcripts never missed a fact but returned a superseded value on every changed fact and leaked every deleted one. Short-term memory alone forgot 36 of 37 facts. A keyed profile with update and delete, plus retrieval over only what the profile does not hold, was the one hybrid that returned the current value on all 74 live probes and leaked nothing.

Start by checking your own retrieval index today. If it holds the raw messages your extractor already turned into memories, remove them, then rerun one question about a fact that has changed. Next, read the site’s guide to context engineering to see what those memories cost once they land in a long agent session.