Write a PREreview

Large language model (LLM) agents that resend an expanding conversation history incur token costs that grow quadratically over a multi-turn task. Many memory designs promise to bound this cost, but their retention/cost trade-offs are frequently measured only on short or synthetic data. This paper is a controlled cost–retention study of seven memory strategies—full context, truncation, running summary, append-only notes, cheap-extractor fact slots, sparse BM25 retrieval, and a slots+retrieval hybrid—evaluated on real, human-annotated long-conversation benchmarks (LoCoMo and LongMemEval) with an LLM-as-judge protocol and question-level paired statistics. Three findings emerge. First, BM25 retrieval is the strongest cheap memory: on LoCoMo (n=160 questions over conversations of 369–680 turns) it cuts real input tokens by 98% and cost to 4.8% of full context while holding retention level with it (39.4% vs. 30.6% point estimates; paired sign test p=0.09, a nonsignificant trend whose confirmation a power analysis places at ≈360–430 questions), and it is significantly better than truncation (p<0.001). Second, we report a negative result with a diagnosis: latest-value fact slots written by a cheap 8B extractor—a structured memory we hypothesized would excel at fact updates—are the weakest memory method on both benchmarks, including on LongMemEval’s knowledgeupdate category (n=72; 37.5% vs. 73.6% for retrieval). An extractor-size ablation isolates the cause: swapping the 8B extractor for a 70B model raises slot retention by +29.2 points (paired sign test, 23 wins vs. 2 losses, p<10−4 ), showing the extractor, not the slot representation, is the bottleneck—but the stronger extractor costs 1.8× full context on these histories, so the cheap-cascade premise itself is what fails. Third, an error analysis over all 281 LoCoMo multi-hop questions shows sparse retrieval recovers only ∼13% of gold evidence turns, localizing exactly where retrieval-based memory still fails. A further LongMemEval evaluation over temporal, multi-session, and abstention questions (n=254) sharpens the boundary of the claim: on these short-history, dispersed-evidence categories full context leads, BM25 trails it significantly (−10.2 points, p=0.004), and only the slots+retrieval hybrid stays statistically indistinguishable from it—so retrieval-based memory is the right trade at long horizons and point lookups, not universally. We release the full harness, benchmark adapters, judge, and per-question results.