PREreview de Efficient Benchmarking in Production: A Study of an Evolving LLM Agent

## Summary

This paper reports first-hand deployment experience with efficient recurring evaluation of a production analytics agent serving tens of thousands of monthly active users. Using 574 historical runs split chronologically into calibration and held-out periods, the authors compare random sampling, historical caching, fixed representative subsets, and IRT-based adaptive testing. Adaptive testing achieves the best fidelity — 200 questions (38.5% of a full run) at 1.03 percentage points of MAE. Despite this, the team deployed difficulty-stratified fixed subsets for operational simplicity, showing they transfer without recalibration to five other agent families. The paper closes with practical recommendations.

## Strengths

First-hand deployment experience is the rarest currency in this literature. The most practitioner-credible admission is that adaptive testing won on fidelity but the team shipped fixed subsets anyway — that is what real engineering trade-offs look like. The chronological calibration/held-out split is methodologically correct, and transfer without recalibration to five other agent families is a strong generalizability claim. Stability across one-day calibration windows means teams can refresh evaluation cheaply.

## Major concerns and questions

1. Is 1.03pp MAE good enough? The paper never connects the error budget to a deployment decision. A worked example tying MAE to a concrete gate threshold would close the loop.

2. What exactly transfers — score fidelity or rank ordering? For a deployment gate, what matters is whether the subset preserves the ship/block decision.

3. What happens under large agent changes — new model generation, new tools, rewritten planner? Some characterization of the subset's "shelf life" would be valuable.

4. All 574 runs come from one analytics agent. A discussion of which workload characteristics make this approach work — and where it might break — would help readers judge applicability.

5. The motivation is cost, but no dollar figures appear. Practitioners need the economic case stated plainly.

## Minor points

- A brief note on the maintenance burden of the IRT item bank as the agent evolves would round out the operational picture.

## Overall assessment

Exactly the kind of paper the field needs: honest about trade-offs, grounded in production data, and methodologically careful. Connecting fidelity numbers to deployment-gate decisions and stating the dollar economics would make it indispensable.

Competing interests

The author declares that they have no competing interests.

Use of Artificial Intelligence (AI)

The author declares that they used generative AI to come up with new ideas for their review.