LoCoMo benchmark methodology correction

Correction: the 93.5% and 78.51% figures previously published here are invalidated. The provider contained per-question answer-key overrides, and a resumed shared-index run lost its session-scoping map, contaminating saved retrieval results. We are documenting the failure modes and rerunning the full 1,540-question corpus under a clean, auditable protocol before publishing a replacement score.

What LoCoMo actually tests

LoCoMo is ten very long synthetic conversations — each roughly 600 turns and 26,000 tokens, spread across many sessions — with question-answer pairs that require pulling a fact out of a specific turn from hundreds of turns earlier. Questions are labeled by type: single-hop (one fact, one turn), multi-hop (join facts across turns or sessions), temporal (dates, durations, weekday arithmetic), adversarial (the honest answer is "that was never stated"), and open-domain world knowledge. It's the most widely cited benchmark for this category of product, which is exactly why its published scores have become a genuine mess.

The number worth a screenshot

On a 200-question balanced sample (40 per category, ten full LoCoMo conversations), with Gemini 3.7 Flash as both the answering and judging model, MemoryBase scored 187/200.

The number that actually matters more

200 questions is a sample. The full LoCoMo set is 1,540 questions across the same ten conversations. We ran all of them — five parallel shards, zero failed questions — with Gemini 3.6 Flash for both answering and judging (3.7 Flash hit its daily quota mid-run; this is the valid full result for the 3.6 pair, not a like-for-like comparison to the 3.7-based 93.5%).

We also caught our own bad number on the way here: an earlier provisional 82.24% was based on only 715 of 1,540 evaluations having finished, not the full corpus. It looked fine until we checked the denominator. We're mentioning it because that exact mistake — a headline number from a partial or convenient sample — is the recurring shape of the disputes described below.

Retrieval isn't the bottleneck anymore. The answer is.

Hit@10 sits at 87–96% across every category — MemoryBase is finding the right evidence almost every time. Single-hop questions, the simplest category, still only score 47.89% accuracy despite87.79% Hit@10. The evidence was in front of the model. The model just didn't use it correctly:

When retrieval hit (Hit@10 true), answer accuracy was 81.52%. When it missed, accuracy fell to 51.61%. That gap is the honest way to read this benchmark right now: the index is close to solved, and exact-span extraction, temporal arithmetic, and evidence-constrained synthesis are the highest-return work left.

What we're doing about it

Two changes are in progress right now, aimed at different layers of the stack. On the retrieval side — the part that ships to every MemoryBase customer, benchmark or not — we're adding a temporal resolver that normalizes relative dates against a reference date before ranking, and a cheap exact-match/entity/time-window candidate union that runs alongside the existing vector+lexical search, so an exact answer-bearing span doesn't get lost in a purely semantic top-k. On the evaluation side, we're moving the answer step to a structured contract — typed answer, evidence IDs, an explicit abstention gate — with per-category routing and subclaim-level verification for multi-hop questions, plus a second, deterministic scoring layer so a retrieval miss and a judge disagreement stop looking like the same failure. Our own target is 90%+ on the full 1,540-question set, in every major category, not just the categories that are already easy.

What "93%" means industry-wide

We didn't have to look far to see why a headline number alone is worth very little in this market. Zep and Mem0 — two of the more visible names in agent memory — are in an open, public dispute over each other's LoCoMo scores, each publishing an audit that the other side's number doesn't hold up:

That's the standard we're holding our own numbers to. Every MemoryBase score in this post comes with its sample size, its answer/judge model, and its category breakdown, and the weaker full-corpus number is the one we're leading with internally, not the one we're hiding. We'll publish the rerun once the retrieval and evaluation changes above land — same 1,540 questions, same disclosure, whichever direction the number moves.