Honest numbers, including the unfavorable ones.

Run context

Benchmark

LongMemEval (ICLR 2025) — oracle split, 500 instances

Retriever

semantic_search() with BAAI/bge-small-en-v1.5 (384-dim, asymmetric with BGE query prefix)

Indexing

Each LongMemEval session flattened to user-turn text, embedded as one memory. No LLM-based extraction.

Hardware

Mac M2 Pro (16 GB), Linux i7-13700K (62 GB + RTX 2080 Ti), Windows Ryzen 7 5800X (64 GB + RX 6600 XT). See hardware table below.

Date

2026-05-31

About the oracle split

Accuracy — LongMemEval oracle

Embedder behavior is identical across hardware, so the accuracy numbers are the same on all three machines. The structurally-bounded scores are listed for transparency, not as a flex.

Latency — by machine

Same benchmark, same 500 instances, run end-to-end on three different boxes. Both x86 machines have GPUs available but unused — see the gaps section.

Cross-machine peer retrieval (measured)

Linux peer p50 stays remarkably flat at ~36 ms as the store grows from 10 to 500 memories — the embedder cost dominates and the linear scan over a few hundred vectors is negligible. Windows peer p50 runs ~8–13 ms higher (~44 ms at small stores, ~49 ms at 500) because the embedder runs on CPU rather than CUDA. Cross-machine overhead vs Mac local: ~21 ms to Linux, ~30 ms to Windows. Day-to-day variance is real: across three runs spanning evening and morning, the same Linux probe returned p50s between 35 ms and 45 ms, tracking Tailscale path conditions. Even at the worst measurement on file, cross-machine retrieval feels instant for any interactive workflow. Where the linear scan starts to matter is characterized below.

Scaling envelope: where the linear scan starts to bite

The retriever is a brute-force cosine scan over every embedding in the store — no ANN index. For typical stores (hundreds to a couple thousand memories) this is fine; the embedder cost dominates and the scan is essentially free. The question is where that stops being true. Same Linux peer, same query, populated stepwise at 1K, 5K, and 10K synthetic memories.

p50 holds at ~40 ms through 5K, then jumps ~15 ms by 10K — the embedder still dominates median latency, but the scan is now contributing visibly. p95 is where the linear cost shows up first: ~50 ms at 1K doubles to ~115 ms at 5K and ~130 ms at 10K, as the tail samples catch the scan doing real work. ANN indexing becomes worth prioritizing somewhere between 5K and 10K memories per user. Below that, the brute-force scan is the right choice — it's simpler, deterministic, and free.

Correctness: recall preserved across the mesh

A separate probe verifies the architecture preserves retrieval accuracy. The peer is populated with 10 known memories, each semantically distinct, then queried 10 times from the Mac via /peer/memory/search with the natural-language question matching each memory. Expected behavior: top-1 is the right memory every time.

This is the expected behavior given that semantic_search is deterministic and the wire transports JSON. The point of this probe is the negative: confirming the architecture introduces no silent corruption, no truncation, no surprise reordering.

Correctness probe was run independently against both Linux and Windows peers. Both returned recall@1=recall@3=recall@10=1.00 across 10 distinct semantic queries, confirming the architecture is symmetric across the mesh — no wire-path corruption, no platform-specific retrieval drift.

By question type (recall_any@10)

Same structural bound as the headline — included for completeness, not as a flex.

Internal Eval B — Potluck custom corpus

42 memories, 25 queries. Exercises the actual product retrieval path (RRF keyword-semantic fusion, Tier-0 pinned injection, cross-project isolation). 2026-05-28 baseline; will be re-published with each release.

LongMemEval-S — the noise-heavy split

By question type (recall_any@10)

Comparison to Mem0 / Letta / Zep

LongMemEval-S now provides a partial point of comparison (see the section above). Other published benchmarks remain non-comparable for the reasons below.

Run context

Runtime

llama-cpp-python 0.3.23 / 0.3.24 (built from source on Linux + Windows for CUDA / Vulkan)

Models

Phi-3-mini 3.8B, Qwen 2.5 7B, Llama 3.1 8B (all Q4_K_M); Llama 3.1 70B (Q3_K_S). Pinned by SHA-256.

Prompts

10 fixed prompts (2 short Q&A, 3 code-gen, 3 summarization, 2 long-context). Seed 42, temp 0.7, top_p 0.9. 3 runs per cell, report median.

Hardware

Same three machines as above. Backends switch per platform: Metal / CUDA / Vulkan.

Date

2026-06-01

Adapter footnote: llama-cpp-python prefix-matches the KV cache across calls. Without an explicit model.reset() between runs, TTFT on run 2+ measures cache hits, not cold prompt evaluation. Documented here because it’s not in the upstream API docs and is a real footgun for benchmark writers.

Machines

Mac

M2 Pro · Metal · 16 GB unified memory

Linux

i7-13700K + RTX 2080 Ti · CUDA 13 · 11 GB VRAM, 62 GB RAM

Windows

Ryzen 7 5800X + RX 6600 XT · Vulkan · 8 GB VRAM, 64 GB RAM

Generation throughput — tokens per second

Median across the 10-prompt suite. Higher is better. The story changes from row to row and column to column — that is the finding.

Three findings worth surfacing

What the memory bench measures and what it doesn't

Known gaps to fix before next publish

1. MCP latency bench: p50 / p95 from memory_search tool-call to result, against a 500-memory store.
2. NVIDIA on Linux via CUDA — enabled 2026-06-01 (driver 580 / CUDA 13.0 on the RTX 2080 Ti). Real but modest: p50 22.2 → 17.8 ms (~20%), p95 34.6 → 20.8 ms (~40%). Not the order-of-magnitude win — same dispatch-bound dynamic as AMD, but less severe since CUDA dispatch overhead is lower. Wall time actually went up (35.6 → 64.6 s) because of GPU init / context setup cost amortized across cold per-instance runs — that overhead disappears in a long-lived production sidecar.
3. AMD on Windows via DirectML — investigated 2026-06-01 (RX 6600 XT). torch-directml runs the bge-small embedder on the GPU with full recall preserved, but p50 is break-even with CPU (17.9 ms vs 18.9 ms). The 33M-parameter model at batch=1 is dispatch-bound, not compute-bound: per-call kernel-dispatch and host↔device transfer roughly equal the CPU forward pass. ROCm/HIP is N/A — gfx1032 is dropped from the current HIP SDK. AMD will become useful again at the inference benchmark, where 7-8B-param models are compute-bound.
4. Cross-machine peer retrieval shipped 2026-06-02 — measured latency + correctness above. Open follow-ups: (a) signed-nonce challenge replacing mesh-IP-as-credential in v0.5, (b) per-project sharing flags instead of all-or-nothing per-machine opt-in, (c) full latency-at-scale measurements against Windows + bigger store sizes (1K, 5K, 10K memories) to characterize the linear-scan ceiling, (d) graceful federation — a peer-aware retriever that automatically merges results from multiple opted-in peers.
5. Approximate-nearest-neighbor indexing. The current linear embedding scan degrades past ~10,000 memories.
6. Publish the Eval B corpus for full methodology transparency.
7. Inference benchmark next runs: a dedicated long-context bench (8K–32K input tokens), batched-inference throughput, multi-user concurrent serving on the same machine, and Llama 70B at quants we haven’t tested (Q4_K_M, Q5_K_M). The cooperative-compute story needs the multi-tenant numbers eventually.
8. Inference hardware extensions: 24 GB + 48 GB cards (e.g. RTX 3090, 4090, A6000) to find the ceiling where a single consumer card runs 70B usably. Apple Silicon higher tiers (M3 Max, M4 Pro) to know the real Mac ceiling.

How to reproduce

Refresh cadence

Re-run before each release tag. This page is updated by hand after a human reviews the numbers — no automated publishing.