Avalilação PREreview de TestSuiteBench: Coding Agents Struggle to Generate Strong Test Suites from Specifications

Thank you for this paper. You build TestSuiteBench from 165 open-source command-line applications, where the agent gets only a natural-language specification and has to write a complete test suite (Sections 1-2). Suite strength is scored by mutants that each violate a single requirement, 74 per application on average (Section 2.2). The best agent, Claude Opus 5, reaches a mutation score of 55.5% with a validity rate of 96.1%, and four of the six agents stay below 40% (Table 2, Section 3.1). I liked the appendix ablation (Figure 13) showing what mutants look like without the "mutate the logic" instruction, since they copy literal values from the target test, and that you name this limitation openly in Section 5.

I read it as a practitioner who runs coding agents every day. Four comments.

1. On valid but weak tests - in Section 3.1 you find validity of 88%-96% against mutation scores of 38%-56%. It matches what I saw in my own runs. In my published experiment (DOI 10.5281/zenodo.22759217) the agent wrote its own tests, and the test gate returned 4,086 tests and 0 failures across all 36 runs. Tests per run ranged from 97 to 127, because I set a minimum but no maximum. So the agent wrote the criterion and then met it. 0 failures does not mean the agent makes no mistakes. In your setup the implementation and the suite come from separate runs and the implementation is graded by the reference tests (Section 2.6). For each agent, of the reference tests that its own implementation fails, how many of those behaviours does the same agent's own suite catch when run against that implementation? That is the situation in practice, where the same agent builds and then tests.

2. On tests that cannot tell versions apart - a check can be honest, meaning it could fail, and still not separate the reference from a broken version. Figure 14d (Appendix A) shows it well: the mutant emits a Zsh completion script when Elvish is requested, and the Opus test does request Elvish completions, but it only checks that the script is long and mentions commands, so the Zsh script passes too. In a preprint on Qeios (DOI 10.32388/0BV3Z8) I argued that a suite of honest checks can prove nothing if none of them changes state between the compared conditions. That is a property of the suite, not of one check. Could you report, per generated suite, the share of valid tests that kill no mutant at all? Given Section 5, where mutants target only a sample of the requirements, such a test may still guard an untargeted requirement, so the number would be an upper bound.

3. On agents that build their own check - Table 3 (Section 3.2) shows Opus created an executable in 149 of 165 instances and mutants in 33 of 165 without being instructed to, while Sol created mutants in 0. Without a check the agent can run, "looks done" is the only signal it has. I give a second, fresh agent the task to disprove the first one's work, treating it as wrong until shown otherwise. It finds what the first missed, because independent agents rarely make the same mistake. Does the Opus mutation score differ between the 33 instances with mutants and the other 132? And have you tried a second agent that gets the specification and the first suite and must write a spec-violating mutant the suite does not kill?

4. On data - The Reproducibility Statement says the full benchmark data and code will be released upon publication, and I see the first page already links a website and a repository (eth-sri/testsuitebench). Could the release include the six generated suites per instance and the per-mutant kill matrix (which valid generated test killed which mutant)? Then readers could answer questions like the ones above without rerunning agents, which at $10.56 per instance for Opus (Table 2) is not cheap for 165 instances. The run data of my 36 coding-agent runs is public on Hugging Face (DOI 10.57967/hf/10366) and on OSF (DOI 10.17605/OSF.IO/5QTWY), so anyone can recompute it. These are open data records, not peer-reviewed publications.

Thank you for the careful write-up.

Competing interests

The review cites the author's own technical report, preprint and open datasets (DOI 10.5281/zenodo.22759217, 10.32388/0BV3Z8, 10.57967/hf/10366, 10.17605/OSF.IO/5QTWY). No connection to the preprint's authors.

Use of Artificial Intelligence (AI)

The author declares that they used generative AI to come up with new ideas for their review.