PREreview del When Coding Agents Outpace Review: Evaluating PRD-Driven Development in Mature Enterprise Repositories

The paper introduces PACE-Bench, 43 production tasks from an enterprise coding-agent platform with 750 expert-audited Code Points built from the final PRD (Sections 1-3). Criteria, allowed alternatives and weights are frozen before any agent is evaluated, and scoring runs in three stages: L0 structural checks, L1 call-graph search and an L2 LLM judge. Across 11 model-scaffold configurations, mean scores on the common OpenHands scaffold span 52.1 to 78.1, and the best complete-delivery rate is 14.0%, 6 of 43 tasks (Section 4.2, Table 2). Incomplete implementation and missing integration together account for 72.0% of pooled weighted loss (Section 4.3). I liked that the criteria are frozen and checked with positive and negative controls (Section 4.4).

I read it as a practitioner who runs coding agents every day. In my company the software development is done by AI agents, so maybe my practical side is useful here. Four comments.

1. On tests and acceptance - Section 1 says passing unit tests does not establish that all PRD conditions are met, and Table 3 reports test-delivery and "Post" (cases with a successful verification after the last edit) outside the functional score. In my own experiment of 36 agent runs the test gate returned 4,086 tests and 0 failures, yet cost between conditions still differed by about a third. The tests were written by the agent itself, so the agent wrote the criterion and then met it, and "all green" told me little (write-up: https://dev.to/arsentev/all-my-agents-tests-were-green-and-they-told-me-nothing-3n9n). Could you cross-tabulate the "Post" cases against complete delivery, to show how often a green check went together with Missing Code Points?

2. On who accepts the work - in Section 3 the agent never sees the Code Points, and Table 5 gives agent-as-a-judge 78.0% agreement with expert consensus. In my company the agent that builds a game for kids never accepts its own game. A fresh agent does, and in one game the fresh checker struck out 57% of the words in its picture vocabulary. My guides say a second, fresh agent told to refute the work finds what the first one missed, and a model sounds equally confident when it is right and when it is wrong (https://arsentev.ai/guides/make-ai-agents-check-each-other). Did you record the agent's own final claim that the PRD was done, and how often did it meet a Missing verdict?

3. On control D - Table 4 (Section 4.4) shows identifier-only patches scoring 4.03, because the judge gave three Partial verdicts for empty bodies. This is the case I see most in practice - the structure is there and looks done, the behaviour is not. Every task needs its own check the agent can run, or "looks done" is the only signal. In one of our games with 500 levels, the judging agent rejected about 55 false findings from the tester agents, so I trust a judge more when it can be shown wrong. Could you add a deterministic L0 check for empty method bodies before L2, and report the three Partials as a known judge failure mode?

4. On repeats - Section 4.5 and Table 11 show that for Qwen3.7-Max three executions score 66.70, 65.44 and 65.37, and execution varies about 3.7 times more than judging (SD 8.77 against 2.36). In my experiment on clearing agent context I ran six policies with six repeats each, 36 runs (DOI 10.5281/zenodo.22759217). One policy came out about a third more expensive (+33.5%, p = 0.002, 0.011 after Holm correction), while "never clear" was not significant. Without repeats I would have read some noise as a result. Repeated executions are reported for Qwen3.7-Max only, so could you repeat them for the models whose means sit close together in Table 2 before ranking them?

Thank you for a clear and useful paper.

Competing interests

Yes: the text cites the reviewer's own technical report (DOI 10.5281/zenodo.22759217), an article on DEV.to and a guide on arsentev.ai, both by the reviewer; no connection to the article's authors.

Use of Artificial Intelligence (AI)

The author declares that they used generative AI to come up with new ideas for their review.