Semantic caching explained: 40-70% token reduction with vector similarity
Agent Brain's semantic cache reuses answers to similar-enough questions instead of paying full token cost every time — here's how the similarity threshold works.
By Agent Software
Why exact-match caching barely helps
A cache that only matches identical text catches almost nothing in a real workflow, because people don't ask the same question the same way twice. "How does the auth flow work" and "walk me through authentication" mean the same thing but share almost no exact text — an exact-match cache treats them as two unrelated, full-cost requests.
What semantic caching does differently
Agent Brain's cache compares the vector embedding of an incoming question against previously answered ones and looks for meaningful similarity, not textual overlap. When a new question is close enough in meaning to something already answered, the cached answer gets reused instead of spending tokens generating a fresh one.
Why the threshold is the whole game
That similarity check runs against a tuned threshold — around 0.85 in Agent Brain's implementation. This number isn't arbitrary: set it too low and the cache starts returning answers to questions that aren't actually close enough, producing subtly wrong results. Set it too high and the cache only catches near-exact duplicates, losing most of the benefit semantic matching was supposed to add.
Finding the right number took deliberate tuning against real question pairs, not a default picked once and left alone.
What this actually saves
In measured use, this approach has reduced token spend on repeated or near-duplicate questions by 40-70%. That's a meaningful chunk of the ongoing cost of running an agent-heavy workflow, especially in teams where the same handful of architectural or "how does X work" questions come up repeatedly across different sessions and different people.
The honest limitation
Semantic similarity is a heuristic, not a guarantee of correctness. A cached answer can become stale if the underlying system it describes changes, and the cache has no built-in way to know that happened unless the memory layer's governance tools (audit and dedupe) are used to keep it current. The savings are real, but they come with an ongoing maintenance responsibility, not a "set once, forget forever" guarantee.
