Escribir un comentario

The paper proposes REM, which manages experience memory for coding agents through three decisions - what to store, when to recall, and what to forget (Sections 3.2 and 4). Each past trajectory becomes a phase-structured document of about 270 tokens (localize, understand, fix, verify), injected at task start. The raw implementation record is injected only after the first failed validation, and evicted after one exposure. On SWE-ContextBench Lite (99 tasks, Claude Sonnet 4.5), REM resolves 27.6% against 24.9% without memory, and cost falls 17% (Sections 5.1 and 5.2). The authors state clearly that the paired difference is not significant (exact McNemar p = 0.22). I liked that all four record policies are reported in Table 3, even though none reaches p < 0.05.

I read it as a practitioner who runs coding agents every day. In my company the software development is done by AI agents, so maybe my practical side is useful here. Four comments.

1. On the 17% saving - Section 5.1 reports 452K falling to 374K cost-weighted tokens per task, with the history never truncated (Section 4). In my own logs of 722 agent sessions on two machines, more than 85% of the modelled cost was context work, cache reads plus cache writes (DOI 10.5281/zenodo.22759216). On a separate set of 168 sessions, 94.4% of billed tokens were re-reading context already sent, a share of tokens, not of the bill (DOI 10.5281/zenodo.22712985). So I expect most of the saving comes from fewer turns re-reading the prefix. Could you report the 452K and 374K split by token type (fresh, cache write, cache read, output) so readers see where the 17% comes from?

2. On the price of eviction - Table 3 (Section 5.4) shows pinned at 353K and evicted at 374K. In my experiment on clearing agent context (36 runs, six policies with six repeats each), clearing after every task came out about a third more expensive (+33.5%, p = 0.002, 0.011 after Holm correction), with the cheapest point around three tasks between clears (DOI 10.5281/zenodo.22759217). The comparison point was chosen after the runs, so I read it with care. Removing context is not free when the prefix is cached. Is the 353K versus 374K difference in Table 3 the cache being rewritten after the record leaves the prefix?

3. On a short document with detail on demand - Table 2 (Section 5.3) shows the phased document of about 270 tokens reached coverage 69.1%, against 45.7% for the oracle summary of about 390 tokens, and the authors conclude organization, not volume, drives uptake. This matches how I keep the instruction file for my agents - short, under about 200 lines, every line tested with a question: if I remove it, will the agent start making mistakes? If not, cross it out. What is needed only rarely goes into skills, not into the permanent file (https://arsentev.ai/guides/claude-md-memory-file). Did you try the document in the system prompt instead of the file the agent reads first (Appendix B), and did the place change coverage?

4. On the struggle gate - Section 3.2 and Appendix B define the trigger as the first genuinely failed validation, with edits to test files filtered out, and Table 3 shows the record injected in 58% of REM runs. In my guides I write that every task needs its own check the agent can run; without it, "looks done" is the only signal the agent has (https://arsentev.ai/guides/best-practices-habits). The gate can only see agents that test. Could you split the 42% of REM runs where the record was never injected into runs that never ran a validation and runs whose validations passed?

Thank you for a clear and useful paper.

Competing interests

The text cites the reviewer's own technical reports (DOI 10.5281/zenodo.22759216, 10.5281/zenodo.22759217, 10.5281/zenodo.22712985) and guides on arsentev.ai; no connection to the article's authors.

Use of Artificial Intelligence (AI)

The author declares that they used generative AI to come up with new ideas for their review.