Escribir un comentario

The paper (arXiv 2610.21914) is a pre-registered study of 465 merged agent-authored pull requests from 246 repositories (Sections 3.2-3.5). A frozen pipeline of four review agents and one fix agent ran every PR with three LLMs, and one mixed model per LLM tests CodeHealth against log total tokens (Sections 3.6-3.9). Each CodeHealth point goes with 3.5% to 6.1% lower token cost (Section 4.2, Table 8), but adding the size of the scored files absorbs the coefficient (Section 4.6.4, Table 15). I liked the handling of the pre-registration: the disclosure that review sessions started before filing, the amendment with a full re-collection of the 67 affected PRs (Section 3.12), and the file-size refit that weakens your own headline.

I read it as a practitioner who runs coding agents every day. In my company the software development is done by AI agents, so maybe my practical side is useful here. Four comments.

1. On heavy tails (Sections 3.9, 4.2) - you note that one pilot PR was about 45% of a tier's mean token total, while in the main collection no PR exceeded 8.2% of a tier's token sum. In my measurement of 722 agent sessions (150,902 model calls, DOI 10.5281/zenodo.22759216), about 3% of sessions gave 80% of the modeled cost. Sessions longer than 200 calls (8% of sessions) gave more than 90% of the money. So in my data the average is driven by a thin tail. What share of each LLM's tokens comes from the top 10% of PRs, and does the CodeHealth slope hold without them?

2. On cache and price (Section 3.8, Table 4, Section 6.3) - the outcome is input plus output tokens, with cached input recorded separately. In the same 722 sessions, more than 85% of the modeled cost was work with context (cache reads and cache writes). In a separate run of 168 sessions, 94.4% of billed tokens were re-reading context already sent (contextburn, DOI 10.5281/zenodo.22712985). That is a share of tokens, not of the bill, because cached reads are the cheapest tokens. So a raw token total and a bill can rank PRs differently. Could you refit H1 with cached input priced at each provider's cache-read rate?

3. On paging long files (Sections 3.6, 5.2, 3.13) - the paper links the extra iterations to the 500-line limit per read call. In my guides I write that quality drops as the agent's context fills, and that heavy reading should go to a subagent so the main conversation stays clean. Since tool results are appended to the message history (Section 3.6), paging the median unhealthy-tier file of 2,146 lines puts every page into history that is re-sent on every later call. From the session records, can you count file-read calls and repeat reads of the same file per tier? That would test the paging explanation directly.

4. On the cost of what (Sections 5.5, 5.6, 6.3) - the study does not assess whether comments are correct or fixes are good. In my guides I write that every task needs its own check the agent can run. Without it, "looks done" is the only signal the agent has. I also use a second, fresh agent whose job is to refute the first one, because AI sounds equally sure when it is right and when it is wrong. For the 27-PR replication subsample (Section 3.11), you could run the repository tests after the fix and have a fresh agent try to refute each comment. Tokens per surviving finding would be a cheap first step toward Section 5.6.

Thank you for a clear and useful paper.

Competing interests

Yes: the text cites the author's own technical reports (DOI 10.5281/zenodo.22759216, 10.5281/zenodo.22712985); no connection to the article's authors.

Use of Artificial Intelligence (AI)

The author declares that they used generative AI to come up with new ideas for their review.