Write a comment

## Summary

This paper does something the agent-cost conversation has badly needed: it measures. The authors run 8 frontier LLMs × 4 runs × 500 SWE-bench-Verified instances under the OpenHands scaffold and characterize where agentic token spend actually goes. They then formalize a second task — pre-execution self-prediction — asking whether the executing agent can forecast its own token cost before starting. The headline results: spend is input-dominated, heavy-tailed, weakly coupled to task success, and model-idiosyncratic; and agents cannot predict their own spend (Pearson r ≤ 0.39, with systematic underestimation bias) even with privileged access to the repository and tools.

## Strengths

**1. Scale and discipline of measurement.** 16,000 agent runs with a fixed scaffold and task set is a serious empirical commitment. Too much of the cost discourse in our field runs on anecdotes and vendor blog posts; this is the opposite — a controlled, repeatable measurement with a public analysis setup.

**2. The negative result on self-prediction is the most important finding.** That agents systematically underestimate their own token spend — even with full context — has direct operational consequences: you cannot delegate budget enforcement to the agent itself. Cost ceilings must live in the orchestration layer, outside the model's discretion. This matches what I have seen in production deployments, where "let the agent manage its own budget" fails in exactly this optimistic direction.

**3. Heavy-tailed, weakly-coupled-to-success spend.** The finding that spend does not track success is the quantitative backbone for a claim practitioners make informally: the most expensive trajectories are often the ones flailing. This justifies spend caps not merely as cost control but as a quality intervention — cutting off a runaway trajectory early saves money *and* avoids shipping the output of a confused agent.

**4. Honest framing as a first measurement.** The authors do not oversell; they position this as the first systematic characterization and leave the intervention work open. That restraint makes the paper more credible, not less.

## Limitations and questions

**1. One scaffold, one domain.** All measurements run under OpenHands on SWE-bench-Verified coding tasks. Scaffold design choices (observation formatting, tool schemas, max-iteration caps) shape the spend distribution, and coding tasks have a particular cost profile. The qualitative findings (input dominance, heavy tails) likely generalize, but the quantitative ones should be read as scaffold-conditioned until replicated elsewhere.

**2. Tokens are not dollars.** The paper measures token consumption, but production cost governance runs on dollars per task — and price ratios between input/output tokens, caching behavior, and model tiers vary widely across providers. A natural extension: re-express the distributions in dollar terms under two or three real pricing schedules, and check whether the "weakly coupled to success" result survives the repricing.

**3. The prediction task deserves a follow-up on intervention.** The paper establishes that agents cannot forecast spend. The operational question is what happens when you *give* them the forecast (or a budget) as an explicit constraint: does spend compress, does success degrade, and does the underestimation bias shrink with an explicit budget in the prompt? That experiment would turn a measurement paper into a design guide.

**4. Where does the heavy tail live?** It would strengthen the paper to characterize *which* instances drive the tail — are they the hardest tasks, the most ambiguous prompts, or specific failure modes (e.g., repeated tool-call loops)? A tail taxonomy would let practitioners target caps where they matter.

## Verdict

A valuable, well-executed measurement paper. The self-prediction negative result alone justifies the read for anyone operating agents in production: budget enforcement belongs in the orchestration layer, not in the agent. I would like to see the scaffold-generality and dollar-repricing extensions, but the core contribution — the first systematic map of where agentic spend goes — stands on its own.

**Recommendation:** A solid contribution; publish as is, with the extensions above as natural follow-up work.

Competing interests

The author declares that they have no competing interests.

Use of Artificial Intelligence (AI)

The author declares that they used generative AI to come up with new ideas for their review.