One model across two computers. Here is what ran.
Model splitting · September 2026 · Earlier measurements below
A 32B model, split across Linux and Mac
On September 29, 2026, Qwen2.5-Coder-32B-Instruct Q3_K_M ran across two owned computers with an 8,192-token context. The main computer used an NVIDIA RTX 2080 Ti with 11 GiB VRAM and 64 GiB system RAM. The worker was an M2 Pro Mac with 16 GiB unified memory. Linux used Ethernet; the Mac used Wi-Fi.
The run used the exact 15,935,048,672-byte model file and development builds of Potluck on both computers. Its native engine was pinned to commit 5a32f7b. Potluck verified model weights on the selected worker before reporting the split ready.
| Where the weights ran | Model weights |
|---|---|
| Linux CPU | 6.23 GiB |
| Linux GPU · CUDA | 7.58 GiB |
| Remote Mac worker | 1.03 GiB |
Two generation requests completed, each returning 128 native completion tokens with the expected model hash. This confirms that the two computers ran parts of the same model and completed inference together.
This was a bounded generation test. It does not establish better coding quality, a speed improvement, or that the model could not run on either computer alone. Part of the model remained on the Linux CPU. A fully wired comparison and the 32B coding-task acceptance run remain outstanding.
Download status: this Linux GPU + Mac setup used development builds. Linux CLI 0.1.7 includes a CPU split engine; Mac CLI 0.1.6 does not include one, and desktop split setup is not yet enabled. Read how model splitting works.
Earlier measurements below. The following runs were recorded in May and June 2026 on the hardware below. They have not been rerun for the current desktop release. They describe those runs, not a guarantee for your hardware or current model catalog.
10.6 ms
P50 · Mac local retrieval
0 / 500
Errors · oracle split, all machines
3
Platforms benchmarked
0.352
recall_all@1 · the meaningful number
Run context
- Benchmark
- LongMemEval (ICLR 2025), oracle split, 500 instances
- Retriever
semantic_search()with BAAI/bge-small-en-v1.5 (384-dim, asymmetric with BGE query prefix)- Indexing
- Each LongMemEval session flattened to user-turn text, embedded as one memory. No LLM-based extraction.
- Hardware
- Mac M2 Pro (16 GB), Linux i7-13700K (62 GB + RTX 2080 Ti), Windows Ryzen 7 5800X (64 GB + RX 6600 XT). See hardware table below.
- Date
- 2026-05-31
About the oracle split
The LongMemEval oracle split only provides the evidence sessions for each question. There is no filler noise. Every instance's haystack equals its answer-session set (verified: 500 of 500). Maximum sessions per instance is 6; median is 2; 176 of 500 instances have exactly 1 session.
That structure means recall_any@10 = 1.00 is structurally guaranteed for any retriever that returns all sessions, with k=10 and a maximum corpus of 6, you cannot miss. The same applies to nDCG@10.
The meaningful number is recall_all@1 = 0.352: across the 324 multi-session instances, the embedder ranks all required sessions at position 1 about 35% of the time. The hard retrieval challenge, filler noise, lives in the LongMemEval-S and -M splits, which require pinning a local generation model. That run is the priority follow-up.
Accuracy: LongMemEval oracle
Embedder behavior is identical across hardware, so the accuracy numbers are the same on all three machines. The structurally-bounded scores are listed for transparency, not as a flex.
| Metric | Value | Notes |
|---|---|---|
| recall_any@10 | 1.00 | Structurally guaranteed on the oracle split (see note) |
| recall_all@10 | 1.00 | Structurally guaranteed on the oracle split |
| recall_all@1 | 0.352 | Meaningful ranking signal across the 324 multi-session instances |
| nDCG@10 | 1.00 | Same structural bound as recall_any@10 |
| Errors | 0 / 500 on every machine | Identical accuracy across all three runs |
Latency: by machine
Same benchmark, same 500 instances, run end-to-end on three different boxes. Both x86 machines have GPUs available but unused, see the gaps section.
| Machine | Spec / backend | p50 | p95 | Wall time |
|---|---|---|---|---|
| Mac M2 Pro | 16 GB unified memory · macOS · Metal backend | 10.6 ms | 35.0 ms | 44.8 s |
| Linux media server | i7-13700K · 62 GB · RTX 2080 Ti · CUDA 13.0 (driver 580) | 17.8 ms | 20.8 ms | 64.6 s |
| Windows workstation | Ryzen 7 5800X · 64 GB · RX 6600 XT (CPU fallback, ROCm not configured) | 18.9 ms | 21.7 ms | 141.4 s |
Cross-machine peer retrieval (measured)
The Mac issues a memory query. The Tailscale-managed WireGuard mesh routes it to a household peer (Linux or Windows). The peer's sidecar runs semantic_search against its own local bge-small embedding store. The result returns over the same tunnel. End-to-end latency below.
The peer endpoint is POST /peer/memory/search, gated behind POTLUCK_PEER_MEMORY_ENABLED=1 on the machine being queried (opt-in, default off). Trust model: mesh-IP reachability is the credential. The peer_access_middleware on the sidecar already rejects non-loopback callers from /memory and all other strictly-local paths; /peer/* is the only surface peers can reach, and within that surface the new endpoint is the only memory path.
| Path | p50 | p95 | min |
|---|---|---|---|
| Mac local (baseline, empty query) | 14.8 ms | 18.3 ms | 7.6 ms |
| Mac → Linux peer (10 memories on peer) | 35.9 ms | 40.7 ms | 18.1 ms |
| Mac → Linux peer (100 memories on peer) | 36.1 ms | 41.2 ms | 19.1 ms |
| Mac → Linux peer (500 memories on peer) | 36.3 ms | 46.0 ms | 20.9 ms |
| Mac → Windows peer (10 memories on peer) | 43.7 ms | 50.7 ms | 39.7 ms |
| Mac → Windows peer (100 memories on peer) | 44.9 ms | 55.6 ms | 39.7 ms |
| Mac → Windows peer (500 memories on peer) | 49.3 ms | 64.3 ms | 45.5 ms |
Linux peer p50 stays remarkably flat at ~36 ms as the store grows from 10 to 500 memories, the embedder cost dominates and the linear scan over a few hundred vectors is negligible. Windows peer p50 runs ~8-13 ms higher (~44 ms at small stores, ~49 ms at 500) because the embedder runs on CPU rather than CUDA. Cross-machine overhead vs Mac local: ~21 ms to Linux, ~30 ms to Windows. Day-to-day variance is real: across three runs spanning evening and morning, the same Linux probe returned p50s between 35 ms and 45 ms, tracking Tailscale path conditions. Even at the worst measurement on file, cross-machine retrieval feels instant for any interactive workflow. Where the linear scan starts to matter is characterized below.
Scaling envelope: where the linear scan starts to bite
The retriever is a brute-force cosine scan over every embedding in the store, no ANN index. For typical stores (hundreds to a couple thousand memories) this is fine; the embedder cost dominates and the scan is essentially free. The question is where that stops being true.
| Path | p50 | p95 | min |
|---|---|---|---|
| Mac → Linux peer (1,000 memories on peer) | 42.0 ms | 51.1 ms | 21.3 ms |
| Mac → Linux peer (5,000 memories on peer) | 40.6 ms | 116.5 ms | 32.0 ms |
| Mac → Linux peer (10,000 memories on peer) | 57.6 ms | 131.9 ms | 47.2 ms |
p50 holds at ~40 ms through 5K, then jumps ~15 ms by 10K, the embedder still dominates median latency, but the scan is now contributing visibly. p95 is where the linear cost shows up first: ~50 ms at 1K doubles to ~115 ms at 5K and ~130 ms at 10K, as the tail samples catch the scan doing real work. ANN indexing becomes worth prioritizing somewhere between 5K and 10K memories per user. Below that, the brute-force scan is the right choice: it's simpler, deterministic, and free.
Correctness: recall preserved across the mesh
A separate probe verifies the architecture preserves retrieval accuracy. The peer is populated with 10 known memories, each semantically distinct, then queried 10 times from the Mac via /peer/memory/search with the natural-language question matching each memory. Expected behavior: top-1 is the right memory every time.
| Metric | Value | Note |
|---|---|---|
| recall@1 | 1.00 | Top-1 hit on every query. The expected memory was always the first result. |
| recall@3 | 1.00 | All 10 queries returned the right memory in their top 3. |
| recall@10 | 1.00 | No misses across 10 distinct semantic queries. |
By question type (recall_any@10)
| Question type | n | recall_any@10 |
|---|---|---|
| knowledge-update | 78 | 1.00 |
| multi-session | 133 | 1.00 |
| single-session-assistant | 56 | 1.00 |
| single-session-preference | 30 | 1.00 |
| single-session-user | 70 | 1.00 |
| temporal-reasoning | 133 | 1.00 |
Internal Eval B: Potluck custom corpus
42 memories, 25 queries. Exercises the actual product retrieval path (RRF keyword-semantic fusion, Tier-0 pinned injection, cross-project isolation). Historical baseline measured May 28, 2026; not rerun for desktop 0.1.6.
| Metric | Value |
|---|---|
| recall@5 (keyword only) | 0.48 |
| recall@5 (RRF, default) | 0.72 |
| Lift from RRF | +0.24 |
| Cross-project leak rate | 0 / 25 queries |
LongMemEval-S: the noise-heavy split
The oracle results above are structurally bounded. The S split is where retrieval actually has to work. Each of the 500 instances includes ~48 sessions (median; range 38-62), with the 1-2 gold-evidence sessions hidden among them. We report retrieval metrics for this split, not end-to-end answer accuracy.
We ran the same retriever (BAAI/bge-small-en-v1.5, 33M params, plain semantic search, no graph extraction, no LLM-based memory extraction) against this split on the Mac M2 Pro.
| Metric | Value | Notes |
|---|---|---|
| recall_any@10 | 0.980 | At least one gold session in top-10 across 500 instances |
| recall_all@10 | 0.938 | All gold sessions for an instance present in top-10 |
| nDCG@10 | 0.885 | Discounted-gain ranking quality |
| p50 latency | 12.9 ms | Per retrieval call; slight bump over oracle (48 sessions vs 3) |
| p95 latency | 39.8 ms | Tail expands modestly with corpus size |
| Errors | 0 / 500 | Wall time 369 s on Mac M2 Pro Metal |
By question type (recall_any@10)
| Question type | n | recall_any@10 |
|---|---|---|
| knowledge-update | 78 | 1.000 |
| multi-session | 133 | 1.000 |
| single-session-assistant | 56 | 0.964 |
| single-session-preference | 30 | 0.933 |
| single-session-user | 70 | 0.986 |
| temporal-reasoning | 133 | 0.962 |
What this number is and what it isn’t.
The 0.98 result measures whether at least one gold-evidence session appears in the top ten retrieved sessions. It does not measure answer accuracy, reasoning quality, or the fraction of all required evidence retrieved.
This retrieval-only experiment is not a head-to-head comparison with systems that evaluate generated answers. A fair comparison needs the same corpus, evidence granularity, metric, and evaluation procedure. We do not claim a performance lead from unlike measurements.
The other caveat: bge-small was trained on broad web data that may overlap with LongMemEval’s academic distribution. Our internal Eval B is the more honest dogfood-distribution check.
Local model inference
Throughput across machines.
Same retrieval workloads were dispatch-bound. Inference at 7-8B parameters is the opposite, compute-bound, where GPUs actually earn their keep. We ran the same four models across the same three machines and the story is materially different.
Correction, October 10, 2026
The generation figures on this page from June were about half the true rate. The benchmark harness generated each answer twice, once to time the first token and once to count tokens, and then divided one answer’s tokens by the time of both. Every platform was low by about the same factor, so the ratios between machines mostly stood. The absolute numbers did not.
The harness now times one pass and reads llama.cpp’s own counters. Linux and Windows have been rerun with it. The Mac column is re-derived from the timings stored with the June run, and the same arithmetic lands within 3% of the measured figures on the other two machines. Time to first token was not affected.
Run context
- Runtime
llama-cpp-python0.3.23 on the Mac (June), 0.3.25 on Linux and 0.3.35 on Windows (October)- Models
- Phi-3-mini 3.8B, Qwen 2.5 7B, Llama 3.1 8B (all Q4_K_M); Llama 3.1 70B (Q3_K_S). Pinned by SHA-256.
- Prompts
- 10 fixed prompts (2 short Q&A, 3 code-gen, 3 summarization, 2 long-context). Seed 42, temp 0.7, top_p 0.9. 3 runs per cell, report median.
- Hardware
- Same three machines as above. Backends switch per platform: Metal / CUDA / Vulkan.
- Date
- June 2026 (Mac). 2026-10-10 (Linux and Windows reruns, and the correction).
Machines
- Mac
- M2 Pro · Metal · 16 GB unified memory
- Linux
- i7-13700K + RTX 2080 Ti · CUDA 13 · 11 GB VRAM, 62 GB RAM
- Windows
- Ryzen 7 5800X + RX 6600 XT · Vulkan · 8 GB VRAM, 64 GB RAM
Generation throughput: tokens per second
Median across the 10-prompt suite. Higher is better. Linux and Windows are measured with the fixed harness. The Mac is re-derived from June and marked ≈.
| Model | Mac | Linux | Windows | Notes |
|---|---|---|---|---|
| Phi-3-mini 3.8B Q4_K_M | ≈ 42 | 146.2 | 34.5 | Published in June as 19.1, 69.2 and 16.2. |
| Qwen 2.5 7B Q4_K_M | ≈ 27 | 96.8 | 48.8 | Published in June as 12.4, 45.9 and 15.9. |
| Llama 3.1 8B Q4_K_M | ≈ 25 | 89.9 | 44.9 | Published in June as 11.6, 43.0 and 2.1. In June the 8B spilled to CPU on the 6600 XT at 8K context. In October it fits the card’s 8 GB. |
| Llama 3.1 70B Q3_K_S | won’t fit | 2.4 | won’t fit | Linux runs with about 20 of 80 layers on GPU, the rest on CPU, at a 2,048-token context. Unusable for chat. A single-prompt measure from June, outside the suite, left as published. |
Four findings worth surfacing
1. There is no universal winner.
The M2 Pro leads on the small Phi-3 mini (about 42 against 34.5 tok/s). The RX 6600 XT leads on the 7B and the 8B (48.8 and 44.9 against about 27 and 25). The Mac figures are re-derived from June and the Windows figures were measured in October on a newer llama.cpp, so the size of each gap is soft. The direction is the one the June runs showed for the models that fit the card.
2. An 8 GB AMD card now runs the 8B model at 8K context.
In June the 8B spilled to CPU on the RX 6600 XT, and this page said an 8 GB card could not hold it. In October the same card runs it fully on the GPU at 44.9 tok/s, with about 6.3 GB in use. Between the two runs the machine got a fresh Windows install, AMD’s August 2026 driver and a newer llama.cpp, and we did not isolate which of them did it. Measured the same day, the RTX 2080 Ti is about 2× the 6600 XT on the 7B (96.8 against 48.8 tok/s) and on the 8B (89.9 against 44.9). In June the ratio on the 7B was 2.9×. The AMD figure has risen since then, and the NVIDIA figure has not moved.
3. A stale GPU driver produced wrong text at full speed.
On the driver Windows installs by itself for this card, dated July 2021, the Vulkan backend repeated words in every reply while the CPU gave correct text. Nothing failed and the speed looked normal. Updating AMD’s driver fixed it. A throughput benchmark alone would not have caught this, and Potluck does not yet check that a GPU backend returns correct text before using it.
4. 70B is the local-only ceiling on this hardware tier.
Llama 3.1 70B at Q3_K_S won’t load in 16 GB Mac unified memory or 8 GB AMD VRAM. On the Linux box with 62 GB system RAM and 11 GB VRAM, it runs with about 20 of its 80 layers offloaded to the GPU and the rest on CPU, at a 2,048-token context. The result is 2.4 tok/s. Technically possible, unusable for chat. The gap between the consumer-hardware ceiling and a usable 70B is roughly 10 to 20×, and a single consumer card does not close it. Pooled compute does.
And 70B isn’t even the local-model frontier in 2026. DeepSeek V3 and R1 (671B MoE, ~37B active) at Unsloth’s most aggressive Q1.58 quant need ~131 GB of unified memory or 192 GB+ of system RAM, a hardware tier above any of the three machines benched here. People do run these models on consumer hardware; they run them on Mac Studio M3 Ultras and 192 GB+ DDR5 workstations. The shared-compute argument isn’t abstract. It’s whichever model your hardware can’t fit.
What the oracle memory bench measures and what it doesn't
It measures:
- Whether the bge-small embedder can rank evidence sessions correctly when given only those sessions.
- Per-call retrieval latency on Apple Silicon (the reference hardware tier).
- Absence of crashes or OOMs on the full 500-instance suite.
It does not measure:
- Retrieval quality against filler noise. The separate S-split results above address that question.
- End-to-end answer accuracy (needs an LLM judge and a generation model).
- The full
get_relevant_memories()path with RRF and ambient-preference injection. Oracle split has no ambient preferences, so this would add noise without measuring anything. - The latency of every hardware/backend combination. See the platform measurements for the specific machines tested.
- Memory store scalability past ~10,000 entries (no ANN index yet).
Known gaps to fix before next publish
- Rerun the Mac column of the inference table with the fixed harness. Linux and Windows were rerun 2026-10-10.
- MCP latency bench: p50 / p95 from memory_search tool-call to result, against a 500-memory store.
- NVIDIA on Linux via CUDA, enabled 2026-06-01 (driver 580 / CUDA 13.0 on the RTX 2080 Ti). Real but modest: p50 22.2 → 17.8 ms (~20%), p95 34.6 → 20.8 ms (~40%). Not the order-of-magnitude win, same dispatch-bound dynamic as AMD, but less severe since CUDA dispatch overhead is lower. Wall time actually went up (35.6 → 64.6 s) because of GPU init / context setup cost amortized across cold per-instance runs; that overhead disappears in a long-lived production sidecar.
- AMD on Windows via DirectML, investigated 2026-06-01 (RX 6600 XT). torch-directml runs the bge-small embedder on the GPU with full recall preserved, but p50 is break-even with CPU (17.9 ms vs 18.9 ms). The 33M-parameter model at batch=1 is dispatch-bound, not compute-bound: per-call kernel-dispatch and host↔device transfer roughly equal the CPU forward pass. ROCm/HIP is N/A; gfx1032 is dropped from the current HIP SDK. AMD will become useful again at the inference benchmark, where 7-8B-param models are compute-bound.
- Cross-machine peer retrieval shipped 2026-06-02, measured latency + correctness above. Open follow-ups: (a) signed-nonce challenge replacing mesh-IP-as-credential, (b) per-project sharing flags instead of all-or-nothing per-machine opt-in, (c) full latency-at-scale measurements against Windows + bigger store sizes (1K, 5K, 10K memories) to characterize the linear-scan ceiling, (d) graceful federation, a peer-aware retriever that automatically merges results from multiple opted-in peers.
- Approximate-nearest-neighbor indexing. The current linear embedding scan degrades past ~10,000 memories.
- Publish the Eval B corpus for full methodology transparency.
- Inference benchmark next runs: a dedicated long-context bench (8K-32K input tokens), batched-inference throughput, multi-user concurrent serving on the same machine, and Llama 70B at quants we haven’t tested (Q4_K_M, Q5_K_M). The shared-compute story needs the multi-tenant numbers eventually.
- Inference hardware extensions: 24 GB + 48 GB cards (e.g. RTX 3090, 4090, A6000) to find the ceiling where a single consumer card runs 70B usably. Apple Silicon higher tiers (M3 Max, M4 Pro) to know the real Mac ceiling.
Recorded results and reproducibility
The application source and benchmark harness are public. These remain first-party measurements; publishing the code does not make them independently reproduced results. Read the recorded LongMemEval-S summary. The commands below document how the runs were made.
make bench from the repo root runs the full 500-instance suite. Results land in bench/results/<timestamp>/. The embedder (BAAI/bge-small-en-v1.5) downloads on first run from HuggingFace (~33 million parameters). No OpenAI API key, no GPU required.
LongMemEval data is used under the original MIT license. Vendored at bench/longmemeval/data/longmemeval_oracle.json. Source: github.com/xiaowu0162/LongMemEval.
For the inference bench: make bench-inference MODEL=bench/models/<name>.gguf. Models are pinned by SHA-256 in bench/models/SHA256SUMS.txt and downloaded on first run via the verification helper. Phi-3-mini-3.8B, Qwen-2.5-7B, and Llama-3.1-8B are ~2-5 GB each at Q4_K_M; Llama-3.1-70B-Q3_K_S is ~29 GB.
Refresh cadence
We update the measurement dates when new runs are reviewed. A desktop release does not automatically mean these benchmarks have been rerun. The model-splitting result is from September 29, 2026. The memory retrieval and local inference results retain their May–June 2026 dates.