Content-hash dedup vs semantic dedup: two different problems, two different fixes
Why Agent Brain uses two separate deduplication strategies instead of relying on similarity search alone to catch duplicate memories.
By Agent Software
Two things get called "duplicate" that aren't the same problem
When people talk about deduplicating a memory store, they usually mean one of two very different situations. The first: the exact same note got saved twice, word for word. The second: the same underlying fact got saved twice, but reworded — a decision restated a week later in different language. Solving one doesn't solve the other.
Content-hash dedup handles the exact case
For byte-identical duplicates, a content hash is the right tool: hash the memory's content, compare it against existing hashes, and reject or merge on a match. It's fast, deterministic, and has zero false positives — if the hashes match, the content is identical.
Semantic dedup handles the reworded case
A hash comparison is blind to rewording, since the underlying bytes are different even when the meaning is identical. Catching that requires embedding the memory and comparing it against existing memories by similarity, using a threshold tuned to catch genuine restatements without merging memories that just happen to share vocabulary.
Why running both matters more than picking one
Relying only on content-hash dedup leaves every rephrased memory uncaught, so a knowledge base slowly fills with near-duplicates that just happen to be worded differently. Relying only on semantic dedup is slower for the common case and risks being too aggressive, occasionally merging memories that are actually distinct but topically similar.
Running the cheap exact check first, and only falling through to the more expensive semantic check when needed, gets the benefit of both without paying the cost of either one alone.
What this looks like in the audit tooling
This two-layer approach is also what powers Agent Brain's memory audit and consolidation tooling — the same distinction between "identical" and "similar enough to review" shows up when deciding what to flag for a human to look at versus what's safe to merge automatically.
