Saltar a PREreview

PREreview del Skip the Compressor: Coding Agents Fix More Issues When They Pick Whole Files

Publicado
DOI
10.5281/zenodo.23468501
Licencia
CC0 1.0

Xiaocong, Rishit, Himanshu, Vikram and ChengXiang, thank you for this paper. It compares ways to fill a coding agent's context at a fixed per-issue token budget: three prompt compressors, three retrievers, two trained methods (SWE-Pruner, SweRank) and the simple selector CQ2-CARE. It is tested on 300 SWE-rebench issues with GPT-5.6 and Claude Opus 5 (Sections 2-3). CQ2-CARE resolves 43.7% and 46.7% of issues, 6.3 points above the best compressor (Table 1). Section 4 traces the gap to retrievers missing the files and to compressors altering the text, so fewer patches apply. I liked that the budget is matched per issue, that the controlled tests change one factor at a time (Section 4.2), and that Appendix C openly reports your own earlier grading mistake.

I read it as a practitioner who runs coding agents every day. In my company the software development is done by AI agents, so maybe my practical side is useful here. Four comments.

1. On cost in a multi-turn loop - Section 4.4 reports that CQ2-CARE uses 0.68 times the tokens with GPT-5.6 and 0.73 times with Claude in one call, and Section 6 leaves the full agent loop to future work. In my measurement of 722 agent sessions (150,902 model calls, DOI 10.5281/zenodo.22759216), more than 85% of the modeled cost was work with context (cache reads and cache writes), not output. In a separate run of 168 sessions, 94.4% of billed tokens were re-reading context already sent (DOI 10.5281/zenodo.22712985). That is a share of tokens, not of the bill. So whatever the selector puts in the context is paid for again on every later call. In the full-loop study, could you report cumulative tokens over all turns, including cache reads, per method?

2. On altered text going unnoticed - Figure 1 shows a pruner deleting the f of an f-string, and the model then patches a line that does not exist. Appendix F adds that 31 of the 36 unparsed Claude answers on LongLLMLingua contexts were placeholders or empty patches. In my guides I write that a model sounds equally confident when it is right and when it is wrong. Every task needs its own check that the agent can run, because without one "looks done" is the only signal it has. A patch on a missing line is exactly that case. In the agent loop, you could give the agent a cheap check, whether its patch applies to the real file, and report per method how many non-applying patches it then fixes on a second try.

3. On repeats - the Limitations and Section 4.3 say most cells have one patch sample, and a second sample flips 8.7-18.9% of outcomes. In my own experiment on clearing agent context I ran six policies with six repeats each, 36 runs (DOI 10.5281/zenodo.22759217). One policy came out about a third more expensive (+33.5%, p = 0.002, 0.011 after Holm correction), while the difference for "never clear" was not significant. Without repeats I would have read some noise as a result. The two-sample average for GPT-5.6 is reassuring. The same second sample for Claude on the narrow contrasts, such as LLMLingua-2 (p = 0.027) and SWE-Pruner, would settle them.

4. On grading - Appendix C says that grading by exit code counted an empty patch as resolved on 4 of 6 checked issues, and Section 3.1 says 328 of 340 candidates passed the eligibility check. In another write-up of the same 36 runs, the test gate returned 4,086 tests and 0 failures, yet cost between conditions still differed. The agent had written those tests itself, so "all green" said little. Your empty-patch check is the cleanest guard against that I have seen. In the released per-issue outcomes, could you include the eligibility check (gold patch resolves, empty patch does not) as its own column, so others can reuse it?

Thank you for a clear and useful paper.

Competing interests

The review cites my own technical reports (DOIs given in the text). I have no relationship with the authors or their institution.

Use of Artificial Intelligence (AI)

The author declares that they used generative AI to come up with new ideas for their review.