Avalilação PREreview de VeriHarness: Scaling Agentic Verification for Long-Horizon Tasks

## Summary

As agents take on long-horizon work — reports, spreadsheets, multi-file artifacts — checking their outputs becomes the bottleneck. This paper studies verification under the hardest honest constraint: no stronger judge exists, so the verifier must use the same model as the generator, with no access to reference answers or grading rubrics. The central empirical finding is striking: when you sample many rollouts of the same task, agreement among them does not reliably mean correctness. On ten-rollout pools from Claude Opus 4.8, about a third of the values all rollouts agreed on were judged wrong, while three-quarters of disputed claims contained a correct candidate somewhere — and the most common answer was right less than half the time. Majority voting is therefore not just weak; it actively preserves errors. VeriHarness turns the generator's own model into an agentic verifier with a workspace, evidence tools, and reusable verification skills: a disagreement resolver that tests competing claims against source files, and a consensus challenger that proposes ways the shared consensus could be wrong. The system is training-free and plug-and-play, achieves the highest selection scores among baselines across five workspace benchmarks and two frontier models, and releases ~26,000 rollouts (produced at over $100,000 in compute) for future research.

## Strengths

The "consensus can conceal errors" finding should change default practice — roughly 34% of unanimous values wrong is a refutation of the sample-and-vote recipe, not a rounding error. The same-model constraint is the honest setting: showing gains with the same weights means the improvement comes from the harness and the environment, a much more deployable claim. The verification record (every claim linked to its check, evidence, and verdict) turns verification into training signal. The skill-evolution result — evolved-from-empty skills outperforming human-authored ones on held-out tasks — suggests verification know-how can be accumulated rather than hand-coded. Releasing a $100K rollout corpus is a real community contribution.

## Major concerns and questions

1. What does verification cost? The harness runs a resolver, a challenger, and an adjudicator — each an agentic loop with tool calls — on top of N generator rollouts. If verification costs 3-5x the generation it protects, the economics only work for high-stakes artifacts. A cost-per-verified-task analysis is the missing number that determines whether this is deployable or academic.

2. Correlated blind spots are the core risk of same-model verification: if the generator misreads a source document, the verifier with the same weights may misread it the same way. On the subset of tasks where the verifier fails, how often is the failure a shared misreading versus a checking error? That decomposition would reveal whether the harness is hitting a correlated-error floor.

3. The headline statistics come from one model on one benchmark (Claude Opus 4.8 on APEX-Agents). A small table of these statistics across all ten model-benchmark cells would materially strengthen the paper's foundation.

4. All five benchmarks produce document artifacts — the friendliest setting for evidence-based verification. A discussion of where the claim-level formalism breaks (multi-turn tool-use, emergent code behavior) would help readers scope the contribution.

5. Skill evolution: if the dev set is narrow, the evolved library may be memorizing benchmark-specific patterns. Cross-benchmark-family transfer (evolve on A, test on B) would be the convincing test.

## Overall assessment

An important paper with a genuinely practice-changing core finding. The same-model constraint makes the results honest, and the two-check decomposition is the right abstraction. The missing pieces are cost accounting and a direct measurement of the correlated-error floor — both addressable, both needed before teams should deploy this.

**Verdict: Recommend with revisions.**

Competing interests

The author declares that they have no competing interests.

Use of Artificial Intelligence (AI)

The author declares that they used generative AI to come up with new ideas for their review.