Avalilação PREreview de The Complexity Trap: Simple Observation Masking Is as Efficient as LLM Summarization for Agent Context Management

## Summary

This paper asks a refreshingly unfashionable question: for managing an agent's growing context window, is the clever approach (LLM summarization of past observations) actually better than the dumb one (masking old observations outright)? Across five model configurations on SWE-bench Verified with the SWE-agent scaffold, with 95% bootstrap confidence intervals, the answer is no — simple observation masking was cheapest in 4 of 5 configurations and matched or beat summarization on solve rate in 4 of 5. The paper's most interesting discovery is the *trajectory-elongation effect*: summarization lengthens trajectories by ~15%, apparently because smoothing over raw observations also smooths over the failure signals that would have ended a bad attempt early. Critic-enhanced summarization, meanwhile, is no better and ~25% more expensive.

## Strengths

**1. A real ablation with uncertainty quantification.** The five-configuration comparison with bootstrap intervals is exactly how this kind of claim should be tested. The result is not "masking felt cheaper" — it is a measured, interval-backed comparison, and the authors resist the temptation to overclaim beyond their configurations.

**2. The trajectory-elongation effect is a genuinely new failure mode.** This is the paper's best contribution. The mechanism is plausible and important: summarization as a lossy compression of *evidence*, not just of tokens — it degrades the agent's ability to recognize its own failure. Anyone building context-management policy should know this effect exists, because it means "smarter compression" can be a pessimization on both cost and quality axes simultaneously.

**3. The 84% observation.** That environment observations constitute ~84% of an average SWE-agent turn is a useful, quotable fact. It explains *why* targeting observations alone is so effective and gives practitioners a prior for where to aim their own context budgets.

**4. Cache-hostility of summarization calls.** The finding that summarization generation is 2.86–7.2% of instance cost *and* cache-hostile is the kind of production-relevant detail that academic papers often omit. In real deployments with prompt caching, the true cost gap between masking and summarization is likely larger than reported here.

**5. Stated limitations are real limitations.** One scaffold, one domain, non-adaptive triggers — the authors name all three. This honesty is to their credit.

## Limitations and questions

**1. The masking window is agent-specific.** The paper finds an optimum of M=10 turns for SWE-agent but notes OpenHands needed M=58 in a probe. That is a wide range, and it means "just mask" is not a turnkey policy — the window is a hyperparameter requiring per-scaffold tuning. The paper would be stronger with even a small theory of what determines the optimum (task horizon? tool-call density? observation verbosity?), so practitioners have something better than grid search.

**2. Non-adaptive triggers are the obvious next step.** The authors flag this themselves. A masking policy that triggers on signals of low information value (repeated similar observations, tool-call loops) rather than a fixed turn count could plausibly beat both baselines. This paper establishes the floor that adaptive work must clear.

**3. Why does summarization elongate trajectories?** The hypothesized mechanism — smoothing over failure signals — deserves a direct test. For example: instrument trajectories for explicit failure-recognition events (error messages, repeated identical actions) and compare their frequency and the agent's response latency under masking vs. summarization. If the mechanism is confirmed, it generalizes beyond context management into a broader principle about evidence preservation in agent loops.

**4. Dollar framing, again.** Costs are reported per instance in what appears to be token-derived terms. For production readers, a translation into $/1,000 tasks under a realistic pricing and caching regime would make the 4-of-5 cost win concrete.

## Verdict

An excellent example of a paper whose negative result is more valuable than a positive one would have been. "Simple masking beats clever summarization" is actionable Monday-morning advice, and the trajectory-elongation effect is a failure mode the field should internalize. The limitations are honest and clearly scoped. I would like to see the adaptive-trigger follow-up, but this paper earns its place as is.

**Recommendation:** A strong contribution; publish as is.

Competing interests

The author declares that they have no competing interests.

Use of Artificial Intelligence (AI)

The author declares that they used generative AI to come up with new ideas for their review.