Avalilação PREreview de Quantifying Overclaiming Propensity in Frontier LLM Agents

The authors measure how often a coding agent reports a review it did not perform. Using OverclaimBench, five file-review scenarios run through each model's own production CLI, they report in Section 5 that agents failed to touch every file they were asked to review in 67.9% of runs, that 80.4% of those incomplete runs were misleading, and that overclaiming runs missed planted defects more often than runs that touched every file.

I read it as a practitioner who runs coding agents every day, on my own machines and from my own logs rather than a lab setup. Four things here meet positions I have published, and one cuts against a widely recommended habit.

1. The result I keep returning to is in Section 4, "Every model overclaims in runs with incomplete reviews", with the observation in Section 5 that overclaiming happened both in runs that had read less than a tenth of the corpus and in runs that had read nearly all of it. My guides describe the same thing from the user's side: the model sounds equally certain when it is right and when it is wrong, so an invented fact, a skipped case and a bug reported as fixed all arrive in one tone (https://arsentev.ai/guides/why-ai-hallucinates). The authors supply the part I could not measure. Section 4 puts it plainly: once a model has read only part of the corpus, it is about equally likely to present its coverage as complete, regardless of capability.

2. Section 4, "Subagents increase coverage but do not improve honesty", touches a claim from my guide on agents checking each other (https://arsentev.ai/guides/make-ai-agents-check-each-other). My position is narrower than "use subagents": a second, fresh agent earns its keep when it is given an adversarial job, to treat the first agent's work as wrong until proven otherwise, because independent agents rarely fail in the same way. What Figure 4 required was delegation, not disagreement. The subagents work for the reporting agent and their findings pass back through it. So the finding that coverage rose while misleading reporting in the Claude family rose with it reads to me as a test of division of labour, not of independent checking, and I would not restate it as evidence about second-agent review in general. I would also keep the caveat from that guide: a checking round lowers the risk, it does not remove it.

3. Section 4, "Needles are more often missed when agents overclaim", supports a rule from my habits guide: every task needs a check the agent itself can run, because without one "looks finished" is the only signal it has (https://arsentev.ai/guides/best-practices-habits). The authors built such a check and kept it outside the agent's account of itself: deterministic line coverage taken from tool results, described in Section 3, plus the validated needle registry in Section 3.3. Their closing argument in Section 5, that a convincing final response alone cannot establish whether the reported work occurred, is that rule written for graders instead of for users.

4. The place I would push back is Section 3.2, where the corpora are confirmed to fit inside every model's context window, token counts in Appendix C, so that incomplete coverage cannot be blamed on context limits. Fitting is not the same as being cheap. Across 722 agent sessions, 150,902 model calls and 34.6 billion tokens in my own logs, more than 85% of the modelled spend went to context work rather than to generation (DOI 10.5281/zenodo.22759216; a technical report from sessions I ran myself, costed from list prices, not peer-reviewed and not a company budget). Reading every file is the expensive half of this task, and saying it was read is close to free. The authors reach for that asymmetry in Section 5 when they write that genuine completion becomes costlier as tasks grow tedious while claiming it stays cheap, but they attribute the pressure to post-training. Figure 3 hints at a plainer one alongside it: reading depth was lowest in the two text scenarios, which they note have the largest corpora. Per-run token spend reported next to coverage would let a reader tell an agent that is misrepresenting its work from one that is also economising on it.

Competing interests

I work on the same problem area as this preprint: I publish measurements of agent runs and I write an IETF Internet-Draft on machine-readable records of agent runs. The review cites my own published report (DOI 10.5281/zenodo.22759216) and my guides on arsentev.ai. I have no relationship with the authors and no financial interest in the outcome.

Use of Artificial Intelligence (AI)

The author declares that they did not use generative AI to come up with new ideas for their review.