Benchmarks & methodology
What 65.0 % means,
and what it doesn't.
We publish one headline number and the full method behind it. This page is the method — how the figure is measured, what it measures, and why you can't line it up against a vendor's number and call it a ranking.
The 65.0 % figure is a retrieval-quality measurement: does Cogitent surface the memory that would let an answering model produce the gold answer? It is measured fully locally, on the full LongMemEval-S split of 500 questions — not a subset, not an easier oracle split.
For scale: Zep's published protocol measures full-context GPT-4o at 60.2 % end-to-end — our fully-local retrieval measurement clears that frontier-model baseline. That is the one legitimate cross-reference here: their own published baseline, not a defeat of their system.
How the figure is measured
The pipeline is deterministic and runs on our own hardware — no cloud retrieval service in the loop:
- Corpus. LongMemEval-S full split — 500 questions across 7 question types (6 retrieval + 1 abstention).
- Retrieval. Postgres pgvector seed-retrieval → MMR diversification (λ = 0.7) → a
jina-reranker-v2cross-encoder → top-5 final chunks. - LLM-judge verdict. A judge model reads the question, the gold answer, and the 5 retrieved chunks and returns yes / partial / no: yes means the chunks would let an answering model produce the gold answer.
It is a retrieval-quality proxy, not an end-to-end question-answering score: we measure whether the right memory is put in front of the model, which is the job Cogitent actually does.
Where we're strong, where we're not
We report per-question-type behavior because the average hides the shape. Some cells are strong; at least one is an open architectural gap we name openly rather than average away.
| Question type | Retrieval@5 | Read |
|---|---|---|
| single-session-user | 80.0 % | Strong — direct facts stated by the user surface reliably. |
| knowledge-update | 74.4 % | Strong — later corrections are found over stale originals. |
| single-session-assistant | 55.4 % | Solid. |
| multi-session | 39.9 % | Improving — date-proximity reranking carries context across sessions. |
| temporal-reasoning | 23.3 % | Hard — reasoning over deltas is a known weak spot. |
| single-session-preference | 0.0 % | Open gap — preference statements aren't yet extracted/surfaced. Architectural, not a compute problem. |
Why you can't rank these against each other
Memory products publish numbers on different measurement bases — different metrics, different splits, different judges, end-to-end QA versus retrieval quality. There is no shared axis to sort them on, so we don't build one. Each card below leads with the basis a system actually measured; the figure appears only inside the sentence that explains it, shown as reported by its authors.
End-to-end QA accuracy
Zep reports 71.2 % end-to-end QA accuracy on LongMemEval-S with GPT-4o answering — a whole-pipeline answer score, not a retrieval measurement, and a protocol whose own full-context GPT-4o baseline scores 60.2 %.
as reported
Fact-level retrieval hit-rate
MemX reports Hit@5 51.6 % at fact-level granularity — a retrieval hit-rate over extracted facts, again a different measure on a different evaluation.
as reported
Vendor's own LLM-judge harness
MemOS (MemTensor) self-reports 89.2 % on LongMemEval under its own LLM-judge harness (88.83 % on LoCoMo) — a vendor-run judge, not one we can reproduce here.
as reported
No comparable figure published
cognee publishes no comparable LongMemEval-S or LoCoMo figure; its self-reported headline is DeepEval Correctness 0.79 on its own BEAM-100K benchmark — a different task on a different dataset.
as reported
Read each card as an example of a different basis. A vendor-reported LLM-judge number is directional information about that system, not a coordinate on a shared axis with our retrieval-quality figure. When we close our own known gaps — preference extraction, temporal reasoning — we'll re-measure and say so, with the method attached.
The number is a floor, not a headline
We'd rather ship one honest figure with its full method than a bigger number you can't check.