Écrire un commentaire

## Summary

Long-running agents observe more than they can keep, so memory is a control problem: under a budget of K items, what stays available? This paper's key insight is that evaluating memory systems confounds two decisions made at different times: the retention decision (online, blind to the future query) and the selection decision (at query time, query-aware). The author demonstrates this by auditing her own earlier evaluation and finding it riddled with defects — mixed access regimes, a template bug, marked distractors — publishing the self-audit openly, which is rare and commendable. She then rebuilds the evaluation as a factorial design: five retention rules crossed with five selection rules, every cell scored on the same 300 seeded episodes. The decomposition is sharp: holding access fixed, query-aware selection buys +15.5 percentage points of recall; the mixed comparison reports +68.7 — meaning over three-quarters of the headline effect is just history access, not query awareness. Under bounded retention, query, dense, and oracle selectors all land exactly on the retention rate. All 319 observed failures in the bounded-recency condition are evictions, not ranking errors. And with recency retention, recall collapses to exactly 0% once the target is 6+ facts in the past. The takeaway is crisp: bounded-memory evaluations must hold access fixed and report retention and selection separately.

## Strengths

The self-audit in the introduction is the bravest section in any paper I have reviewed this year — every defect found there is currently sitting undetected in someone else's benchmark. The factorial design is executed with unusual statistical care: same seeded episodes in every cell, bootstrap confidence intervals over episodes, per-query records saved. The exact-equality result (selection methods all equal to the retention rate to the decimal) is the kind of clean empirical fact that settles an argument. The "0% at long delay" result converts an abstract point into something a practitioner feels: no retriever can rank an item that no longer exists.

## Major concerns and questions

1. The environment is extremely synthetic — every fact has the identical surface form "The slot is value." The paper's central claim (retention dominates selection) is most convincing exactly where it is least tested. A replication on genuine agent trajectories, even small, would transform this from a methodology parable into an empirical fact about agents.

2. The paper diagnoses but doesn't prescribe: no retention improvement is tested. Even one learned or heuristic retention method outperforming recency would give the paper a constructive half to match its critical half.

3. The privileged type-oracle (70.3%) sits in the same tables as implementable methods and will be the number casual readers remember, though no real system can reach it. Consider moving it to a separate analysis.

4. Table 4 shows higher distractor density benefits the privileged oracle — a warning sign about the environment's structure that deserves a sentence on whether the distractor model was validated against anything.

## Overall assessment

A model methodology paper: honest about its own past errors, statistically careful, and carrying one result (the retention ceiling) that should change how the field evaluates memory. The synthetic environment is the real limitation. Every benchmark builder should read the introduction's self-audit.

**Verdict: Recommend with revisions.**

Competing interests

The author declares that they have no competing interests.

Use of Artificial Intelligence (AI)

The author declares that they used generative AI to come up with new ideas for their review.