past.dev, a San Francisco startup, posted its launch video on X on Oct. 151. Mehdi Djabri, its chief executive, wrote that the service is "#1 on BEAM, the largest public memory benchmark"1. By Oct. 16 the post had 362,813 views and 584 bookmarks1. The claim rests on a run past.dev made on Sept. 293. It scored 92.08% at 100,000 tokens of history, 90.65% at 1 million and 85.03% at 10 million. Four rivals had each published a BEAM table with themselves on top since April11131417. The AILately.com Desk read every table and graded nine memory layers.
How the Grades Work
Each product gets one letter. Four things set it. Evidence is the best BEAM score the vendor published at 10 million tokens, and whether its per-question results are public. Scrutiny asks whether anyone outside the company checked the number. Control covers deletion, retention and who may read what. Openness covers the license and self-hosting. Price enters as a fact, since every plan was read on Oct. 16.
What past.dev Sells
The API has four calls4. An app posts text with the time it happened, then polls until processing ends4. It deletes a record when needed, and it recalls facts with a question and a reader's identity. Each record names an audience, so a sales agent and a support agent asking the same question get different answers4. Every answer carries the dated source behind it, and a recall can read memory as it stood on a past date4. "Deleting a record removes the memory derived from it," past.dev wrote on Sept. 304.
Its engine is closed. Plugins for Claude Code, Codex, Cursor, JetBrains and Hermes are MIT licensed, and each sends only the prompts and replies of a session7. Two MCP servers connect Claude, Cursor, VS Code, Windsurf and Zed6. The "single prompt" in the launch post is a setup file an agent reads, which stops twice for approval1.
Pay as you go costs $0 and starts with 150,000 credits, or 400,000 with a work email5. One credit buys 350 bytes of ingested text. Flex costs $99 a month for 400,000 credits and unlimited recall, down from $299 on Sept. 2958. Retention runs 365 days on the free plan and three years on Flex5. The company says it holds SOC 2 Type II, ISO 27001 and ISO 27701 certificates2. Its API reached early access on Aug. 68.
Five Claims on One Benchmark
BEAM comes from a paper by Mohammad Tavakoli and five co-authors, posted on Oct. 31, 202510. It holds 100 conversations and 2,000 validated questions, at lengths up to 10 million tokens10. The authors found that "even LLMs with 1M token context windows (with and without retrieval-augmentation) struggle as dialogues lengthen"10. Its own retrieval baseline scored 24.9% at 10 million tokens11.
Hindsight, the open-source memory from Vectorize, went first. "Hindsight scores 64.1% at 10M. The next-best published result is 40.6%," Ben Bartholomew wrote on April 211. That 40.6% belonged to Honcho, from Plastic Labs, whose page still says "Top scores across all BEAM tests"13. Exabase, a London company, said on July 28 that its M-1 engine scored 68.0% at 10 million, "surpassing all previously reported systems"14. SurrealDB published its own run on Oct. 14, at the head of its own table17. On Sept. 30, past.dev's 85.03% landed 17 points above Exabase34.
past.dev's 85.03% at 10 million tokens sits 17 points above Exabase's 68.0%
Best BEAM score each vendor published at 10 million tokens of history, percent correct; each vendor ran its own harness; higher is better
Each bar is one vendor's own published BEAM score at the 10-million-token size; the magenta bar is past.dev, and the gray bar is the paper's retrieval baseline.
Sources: past.dev [3]; Hindsight [11]; The AI Journal [14]; Mem0 [16]. Chart by The AILately.com Desk.
The numbers behind this chart
| Item | Value |
|---|---|
| past.dev | 85.0% |
| Exabase M-1 | 68.0% |
| Hindsight | 64.1% |
| Mem0 | 48.6% |
| Honcho | 40.6% |
| Paper's retrieval baseline | 24.9% |
Four vendors published BEAM tables with themselves on top between April 2 and Oct. 14, 2026
Dates on which each memory vendor published a BEAM result that led its own comparison table, 2026
Each point is the day one vendor published a table with its own memory system at the top; the magenta point is past.dev's Sept. 30 post.
Sources: past.dev [4]; Hindsight [11]; The AI Journal [14]; SurrealDB [17]. Chart by The AILately.com Desk.
The numbers behind this chart
| Date | Event |
|---|---|
| April 2, 2026 | Hindsight claims 64.1% at 10M |
| July 28, 2026 | Exabase claims 68.0% at 10M |
| Sept. 30, 2026 | past.dev claims 85.03% at 10M |
| Oct. 14, 2026 | SurrealDB publishes its own run |
The press release past.dev issued on Oct. 6 contains two sets of prior records. Its lead paragraph says the previous bests were 76.9%, 71.1%, 75.0% and 68.0%2. The table below it says 86.2%, 80.1%, 79.1% and 68.0%2. Those first three figures are credited to Hindsight, whose own April post lists 73.4%, 71.1% and 73.9%311.
What the Numbers Leave Out
Each vendor ran its own harness. past.dev says so itself: "These are separately published runs rather than one harness, so models, judges and configurations differ between rows"3. In its run, "One model, openai/gpt-5.6-luna, both answers and judges"3. Mem0's own benchmark guide carries the same warning. "Read the methodology column before the score column: several of these numbers are disputed by the vendors named next to them, and at least one vendor has published two different scores for itself in different posts," Himanshu Sangshetti wrote16.
Tobie Morgan Hitchcock of SurrealDB re-scored four rivals' published answer files at 1 million tokens, counting only full marks17. "Every agent memory vendor publishes a benchmark score. While most of these scores are true, very few of them measure the same thing," he wrote17. Under his strict count, Exabase fell from 75.0% to 48.0% and Hindsight from 73.9% to 48.4%17. Mem0 fell from 64.1% to 40.1%, and Honcho from 63.1% to 41.4%. SurrealDB's own run scored 77.2% and 57.1%17. past.dev's run came too late for his table.
Counting full marks only, Exabase drops from 75.0% to 48.0% and Hindsight from 73.9% to 48.4%
BEAM at 1 million tokens: each vendor's headline score against SurrealDB's strict recount, percent; higher is better
Each row is one memory system: the blue dot is the score the vendor published, and the magenta dot is SurrealDB's recount of the same answers awarding credit for full marks only.
Source: SurrealDB [17]. Chart by The AILately.com Desk.
The numbers behind this chart
| Item | Vendor's headline score | Strict rescore by SurrealDB |
|---|---|---|
| SurrealDB (its own run) | 77.2% | 57.1% |
| Exabase M-1 | 75.0% | 48.0% |
| Hindsight | 73.9% | 48.4% |
| Mem0 | 64.1% | 40.1% |
| Honcho | 63.1% | 41.4% |
LoCoMo, the older benchmark most vendors cite, has its own problem. Penfield Labs audited its answer key in March and found 99 errors in 1,540 questions, or 6.4%18. The same judge accepted 62.81% of answers written to be wrong18. past.dev scored 93.12% on a corrected key3.
Does a Layer Help at All
Mem0 built DolphinBench to answer that, with 600 tasks that an agent must act on19. On the Hermes agent with GPT-5.6-Luna, the agent's built-in memory completed 65.67% of tasks at a total cost of $61.4819. Mem0 raised that to 70.67% at $96.2119. Hindsight reached 69.50% at $84.65, Honcho 68.50% at $142.85 and Supermemory 59.17% at $345.6419. On Claude Code with Claude Sonnet 5, built-in memory scored 26.33% and Mem0 32.33%19.
Memory layers lift a Hermes agent from 65.67% to at most 70.67% of DolphinBench tasks
Share of 600 DolphinBench tasks completed on the Hermes agent with GPT-5.6-Luna, percent, Mem0's benchmark; higher is better
Each column is one memory setup on the same agent and model; the gray column is the agent's built-in memory alone, and the others add one vendor's layer.
Source: Mem0 [19]. Chart by The AILately.com Desk.
The numbers behind this chart
| Item | Value |
|---|---|
| Built-in memory | 65.67% |
| Mem0 | 70.67% |
| Hindsight | 69.50% |
| Honcho | 68.50% |
| Supermemory | 59.17% |
past.dev ran the same tasks itself and reported 519 of 599 done right, against 323 for Claude Code alone4. Its page calls that figure "not an official score"4.
The Grades
| Product | Grade | Entry price | BEAM at 10M | Open source | Why |
|---|---|---|---|---|---|
| past.dev | B+ | $0, Flex $99 | 85.03% | Closed engine, MIT plugins | Highest scores, public per-question results, dated sources and audiences. One model judges its own answers, and the API is ten weeks old38. |
| Hindsight | B+ | Free self-hosted; $10 per million tokens stored | 64.1% | MIT | Runs locally, publishes its harness and an open leaderboard. Its strict score is 48.4%111729. |
| Mem0 | B | Free; Starter $19 | 48.6% | Apache 2.0 | Most used, with 62,590 GitHub stars and $24 million raised2026. Weakest BEAM score of the five, and its own DolphinBench shows a five-point gain at 1.6 times the cost1619. |
| Exabase | B- | Free credit; Scale $149 | 68.0% | Closed | Second-best score with a cheaper model. Strict rescoring gives 48.0%151730. |
| Honcho | B- | Per-call, from $0.001 | 40.6% | AGPL 3.0 | Open and cheap. Its page still claims the top BEAM scores, and "Peers and individual messages cannot be deleted"1332. |
| Zep | C+ | Free; Flex $125 | Unpublished | Apache 2.0 for Graphiti | Dates every fact in a graph and holds SOC 2 Type II. It claims 94.7% on LoCoMo, a number Mem0's guide lists beside a third-party 75.1%162425. |
| Letta | C+ | Free; Pro $20 | Unpublished | Apache 2.0 | Argues memory belongs in the agent, and scored 74.0% on LoCoMo with plain files2327. |
| Supermemory | C | Free; Pro $19 | Unpublished | MIT repo, closed model | Last on DolphinBench at the highest cost, and its 97% LongMemEval figure measures retrieval recall1928. |
| Engram | Incomplete | Unpublished | Unpublished | Closed | Raised $98 million at a $600 million valuation in June, with Microsoft, Notion and Harvey as partners21. A benchmark and a price list have yet to appear. |
The Case For and the Case Against
Djabri argues that search was the wrong tool for memory. "For two years, the industry has tried to solve memory with search. But search finds what looks similar, not what's still true," he said in the Oct. 6 release2. Alex Ronse, his co-founder and chief technology officer, named the hard part. "Deciding that one fact replaced another, across sources, without leaking a private email, is the hard part," Ronse said. "We've opened our evaluation harness so anyone can check our work."2. The harness is public, and the judge inside it is the same model that answers39.
Sarah Wooders, co-founder of Letta, makes the case against buying a layer at all. "RAG over past session data (or a processed form of it) can certainly be a plugin—but retrieval is a small part of memory. And even then, it's hard to do much better than just grep," Wooders wrote on April 322. Her company sells agents, so she has a stake too. Writer's June study gives her side a number31. On moral questions, Claude Sonnet 4.6 moved from a 1.6% sycophancy rate with plain chat history to 40.2% under Mem0, a 25-fold rise.
What Each Side Needs
Djabri's case needs an outside run of past.dev's harness with a different judge model. It also needs a strict rescore in the SurrealDB style. Wooders's case needs the built-in memory of agents to close the five-point DolphinBench gap at lower cost19. Both sides need BEAM's authors to publish a leaderboard, since the only ones today belong to vendors113.
What to watch
- A strict rescore of past.dev's 10M answer file, which its harness makes possible917.
- Mem0's next DolphinBench round, with past.dev and Exabase added19.
- Engram's first published benchmark or price list21.
- The Claude connector directory, where past.dev's docs say it is listed6.
Sources
- Mehdi Djabri, "Introducing past.dev," X, Oct. 15, 2026, https://x.com/mehdidjabri/status/2110762633494184248
- past.dev, "past.dev Introduces Frontier Memory for AI Agents: #1 on BEAM, the Largest Public Memory Benchmark," PR Newswire, Oct. 6, 2026, https://www.prnewswire.com/news-releases/pastdev-introduces-frontier-memory-for-ai-agents-1-on-beam-the-largest-public-memory-benchmark-302899770.html
- past.dev, "Benchmarks," past.dev, Oct. 2, 2026, https://past.dev/benchmarks
- past.dev, "Introducing past.dev: memory for AI agents," past.dev, Sept. 30, 2026, https://past.dev/blog/introducing-past-dev
- past.dev, "Pricing," past.dev, Oct. 16, 2026, https://past.dev/pricing
- past.dev, "What the past.dev MCP servers are," past.dev, Oct. 16, 2026, https://past.dev/docs/mcp/overview
- past.dev, "Agent plugins," past.dev, Oct. 16, 2026, https://past.dev/docs/plugins/overview
- past.dev, "Changelog," past.dev, Oct. 16, 2026, https://past.dev/changelog
- past.dev, "BEAM on past.dev (benchmark harness README)," GitHub, Oct. 16, 2026, https://github.com/pastdotdev/benchmarks/tree/main/beam
- Mohammad Tavakoli, Alireza Salemi, Carrie Ye, Mohamed Abdalla, Hamed Zamani and J Ross Mitchell, "Beyond a Million Tokens: Benchmarking and Enhancing Long-Term Memory in LLMs," arXiv, Oct. 31, 2025, https://arxiv.org/abs/2510.27246
- Ben Bartholomew, "Hindsight Is #1 on BEAM — the Benchmark That Tests Memory at 10 Million Tokens," Hindsight, April 2, 2026, https://hindsight.vectorize.io/blog/2026/04/02/beam-sota
- Nicolò Boschi, "Agent Memory Benchmark: A Manifesto," Hindsight, March 23, 2026, https://hindsight.vectorize.io/blog/2026/03/23/agent-memory-benchmark
- Plastic Labs, "AI memory benchmarks: Honcho on LongMem, LoCoMo, BEAM," Honcho, Oct. 16, 2026, https://honcho.dev/evals/
- Exabase, "Exabase Achieves State of the Art on BEAM Memory Benchmark at Every Scale, Using a Smaller, Cheaper Model," The AI Journal, July 28, 2026, https://aijourn.com/exabase-achieves-state-of-the-art-on-beam-memory-benchmark-at-every-scale-using-a-smaller-cheaper-model/
- Jonathan Bree, "M-1 achieves state-of-the-art on BEAM at every scale, with a smaller model and fewer tokens," Exabase, Oct. 16, 2026, https://exabase.io/blog/exabase-m1-achieves-state-of-the-art-on-beam-benchmark
- Himanshu Sangshetti, "AI Memory Benchmarks: LoCoMo vs. LongMemEval vs. BEAM," Mem0, Sept. 22, 2026, https://mem0.ai/blog/ai-memory-benchmarks-in-2026
- Tobie Morgan Hitchcock, "Agent memory benchmarks are baked," SurrealDB, Oct. 14, 2026, https://surrealdb.com/blog/agent-memory-benchmarks-are-baked
- Penfield Labs, "We audited LoCoMo: 6.4% of the answer key is wrong and the judge accepts up to 63% of intentionally wrong answers," DEV Community, April 4, 2026, https://dev.to/penfieldlabs/we-audited-locomo-64-of-the-answer-key-is-wrong-and-the-judge-accepts-up-to-63-of-intentionally-33lg
- Mem0, "Introducing DolphinBench: Mapping the Pareto Frontier of Agent Memory," Mem0, Sept. 22, 2026, https://mem0.ai/blog/introducing-dolphinbench-mapping-the-pareto-frontier-of-agent-memory
- Tage Kene-Okafor, "Mem0 raises $24M from YC, Peak XV and Basis Set to build the memory layer for AI apps," TechCrunch, Oct. 28, 2025, https://techcrunch.com/2025/10/28/mem0-raises-24m-from-yc-peak-xv-and-basis-set-to-build-the-memory-layer-for-ai-apps/
- Sheryl Hodge, "Engram Launches With $98M to Give Enterprise AI Memory," VKTR, June 24, 2026, https://www.vktr.com/ai-news/engram-launches-with-98m-to-give-enterprise-ai-memory/
- Sarah Wooders, "Why Memory Isn't a Plugin," Letta, April 3, 2026, https://www.letta.com/blog/why-memory-isnt-a-plugin/
- Letta, "Benchmarking AI Agent Memory: Is a Filesystem All You Need?," Letta, Aug. 12, 2025, https://www.letta.com/blog/benchmarking-ai-agent-memory/
- Zep, "Graphiti," Zep, Oct. 16, 2026, https://www.getzep.com/platform/graphiti/
- Zep, "Pricing," Zep, Oct. 16, 2026, https://www.getzep.com/pricing/
- Mem0, "Pricing," Mem0, Oct. 16, 2026, https://mem0.ai/pricing
- Letta, "Pricing," Letta, Oct. 16, 2026, https://docs.letta.com/pricing
- supermemory, "Pricing," supermemory, Oct. 16, 2026, https://supermemory.ai/pricing/
- Vectorize, "Pricing," Vectorize, Oct. 16, 2026, https://vectorize.io/pricing
- Exabase, "Pricing," Exabase, Oct. 16, 2026, https://exabase.io/pricing
- Writer, "How personalized context quietly degrades AI accuracy: a deeper look," Writer, June 10, 2026, https://writer.com/engineering/personalized-context-degrades-ai-accuracy/
- Plastic Labs, "Deleting data," Honcho, Oct. 16, 2026, https://honcho.dev/docs/v3/documentation/features/advanced/deleting-data