Escrever um comentário

## Summary

The authors (Meta) present REAP, an automated pipeline that curates execution-based coding-agent benchmarks from real developer-agent sessions, without manual labeling. The motivation is one I live daily: public benchmarks diverge from production workloads in language distribution, prompt style, codebase structure, task mix, and contamination risk, while online A/B testing is slow and shadow deployment is not reproducible. REAP applies three verification filters — LLM-based task classification, agentic test-relevance validation, and multi-run stability checks — to turn noisy production sessions into trustworthy fail-to-pass tasks.

Using REAP they curate Harvest, a benchmark of verbatim developer prompts paired with executable tests, spanning 4+ languages with a Hack majority (a real divergence from Python-skewed public benchmarks). Five frontier models score 42.9% to 58.2%, surfacing capability differences the authors say informed concrete deployment decisions. The paper is refreshingly specific about monorepo constraints: ephemeral build infrastructure, expiring toolchains and indices, and the "time travel" problem that makes rolling re-curation a prerequisite rather than a convenience. Accepted at the ASE 2026 Industry Showcase. The source corpus cannot be released; the authors share methodology, classifier prompts, and annotation protocols instead.

## Strengths

1. Solves the right problem. Verbatim prompts, one-to-one prompt-to-diff mapping via AI-provenance instrumentation, and execution-based scoring with no LLM judges — this is how you build an eval that production teams can actually gate deployments on. The explicit rejection of LLM-judge scoring deserves emphasis.

2. Rolling re-curation is the honest answer to benchmark staleness. The paper's argument that manual verification cannot be sustained at monorepo cadence, so automation is a prerequisite, is well made and matches industrial reality.

3. The test-relevance filter is a real contribution. Naively running CI per candidate is computationally infeasible in a monorepo; selecting a small set of causally-connected, stable fail-to-pass tests is the load-bearing engineering of this work, and the near-human classifier accuracy validation is a solid check.

4. Clear-eyed about the contamination problem. Frozen benchmarks leak into training data; a continuously re-curated benchmark that post-dates model cutoffs is one of the few structural fixes available.

## Major comments

1. The corpus cannot be released, so the headline results are unverifiable. The authors share methodology and prompts but not Harvest itself. I understand the proprietary constraint, but it means the 42.9–58.2% solve rates and the deployment decisions they "informed" rest entirely on trust. At minimum, the paper should release a fully worked synthetic or open-source analogue so the pipeline's output quality can be independently inspected.

2. Scope is narrow for the generality of the claims. The case study is one production assistant, single-turn conversations only, and a Hack-majority task pool. Multi-turn sessions — where agents actually fail in production — are excluded by a "deliberate scope choice." The paper should either demonstrate a multi-turn slice or clearly fence the claims.

3. Agent-run stability is not measured. The pipeline checks test flakiness across runs, but agents are non-deterministic: pass@1 from a single run per model is a point estimate with unknown variance. A benchmark meant for deployment gating needs per-task run-to-run variance of the agent itself, not just of the tests.

4. The agentic test-relevance validator is itself an LLM judgment. The paper validates classifiers against human judgment, which is good — but the "causal connection" between test and diff is the most subtle judgment in the pipeline, and false positives here silently corrupt the benchmark's signal. Report the validator's precision/recall on a human-adjudicated sample explicitly.

## Minor comments

1. The paper's title changed across versions (ProdCodeBench to REAP); the final framing as a pipeline paper is the right one — Harvest is an instance, not the contribution.

2. A cost accounting of the curation pipeline itself (compute per admitted task) would help other organizations budget for adoption.

## Overall assessment

Recommend with revisions. The methodology is strong and the industrial grounding is real — this is one of the few benchmark papers written by people who have felt the pain of evals that don't transfer. But a non-releasable corpus plus single-run agent scores means the deployment-relevant claims need tighter fencing before this can serve as a template others trust.

Competing interests

The author declares that they have no competing interests.

Use of Artificial Intelligence (AI)

The author declares that they used generative AI to come up with new ideas for their review.