PLATE · Ⅲ Benchmarks Public dataset · official scoring · deviations disclosed

Honest benchmarks

The AI-memory market publishes leaderboard numbers that don't survive independent reproduction. We do it differently: every Mnem figure on this page is read straight from a benchmark run's output — nothing is typed by hand — on the public LongMemEval dataset, using the benchmark's own prompts and scoring protocol, with every deviation from that protocol written down here. Recall runs entirely on the device; the only cloud component anywhere below is the benchmark's own judge model, and it is pinned.

Verified performance generated from harness output

†1

LongMemEval end-to-end answer accuracy over all 500 questions — the figure the market quotes — with the official reader and judge prompts, a gpt-4o reader, and the benchmark's own judge pinned.

LongMemEval_s · QA ·

†2

Retrieval Recall@5 on the same dataset: a labeled evidence session sits in Mnem's top-5 recalled sessions. No answering model, no judge — the layer we ship, measured alone.

LongMemEval_s · retrieval · questions

†3

Retrieval Recall@10 — the context window the end-to-end reader actually receives. The gap to 100% is the ceiling on the QA number above.

LongMemEval_s · retrieval · top-10

†4

On-device recall latency at p95, measured through the shipped search path with no network round-trip. The column that decides product fit.

Local harness · recall p95 ·

† Method notes — what was measured, and how

Our rules

Loading…

LongMemEval, end to end measured, not claimed

Question type	n	Accuracy

How it was run: Mnem's shipped recall retrieves the context for each question; a reader model answers from that context using the benchmark's official reader prompt, and the benchmark's official judge prompts grade each answer. Reader and judge models are recorded with the result. The reader and judge prompts are verbatim copies of the benchmark's own, and answers are written in the official layout so the benchmark's own grader scores them unchanged.

Reading it against vendor claims

LongMemEval, retrieval stage measured, not claimed

Question type	n	any@1	any@5	any@10

How it was run: the public LongMemEval dataset is loaded into a fresh Mnem store and every question is answered by the same recall path the app ships, on the device, with no network. The run refuses to proceed unless the semantic encoder is actually loaded — it never silently falls back to the non-semantic encoder.

Published reference points on the same dataset

System	Metric	Figure

Mnem, measured locally measured, not claimed

Category	Items	Recall@1	Recall@5
Loading results…

How it was run: a seeded store with interleaved distractors, queried through the same recall path the app ships, on the device, with no network. Dataset, scorer and configuration are recorded with each result. The run refuses to proceed unless the semantic encoder is actually loaded — it never silently falls back to the non-semantic encoder.

What competitors claim vs. what reproductions found

These are not our measurements. Claims are the vendors' own self-reported figures; reproductions are third-party runs (some by competing vendors — noted where known). We list both because, as of 2026, cross-vendor leaderboard numbers are harness-relative and not comparable. The one axis they share with us is the end-to-end LongMemEval figure above — read the caveats there first.

The columns that decide product fit

	Where memories live	Recall path	Portability

Competitor claims and reproductions cited as of the date in each entry. Nothing on this page is hand-edited: figures are generated from harness output.

Run the numbers yourself.

Sub-10ms recall on your machine, with nothing leaving it.

Get Mnem Read the docs