PREreview of Maximising Transparency in Context Compaction Improves Retention of Facts and Commitments for Lightweight Coding Agents

The paper introduces transparent-compact, a deterministic compaction extension for the open-source Pi coding agent (Section 3.1). It keeps every user message and all assistant prose word for word, reduces tool calls to one-line signatures, removes tool outputs and reasoning, declares what was removed, and recovers it through one delegated lookup subagent, context_lookup. It is compared with Pi's model-written summary (pi-default) and with pi-vcc, a deterministic extractor (Sections 2 and 3.2), on 27 commitment probes from 18 real sessions and 17 planted facts plus 3 lures in 18 synthetic sessions (Section 3.3, Table 2). On planted facts it recalls 97.8% against 75.3% for pi-default on Qwen, 95.1% against 55.0% on Luna and 93.3% against 51.5% on Gemma (Section 4.2, Table 5). On commitments it beats pi-vcc on every model, by 21 points on Qwen (Section 4.1). I liked that failures are split into passes, failures that flagged doubt, and confident failures (Section 4.1, Figure 2).

I read it as a practitioner who runs coding agents every day. In my company the software development is done by AI agents, so maybe my practical side is useful here. Four comments.

1. On cost (Section 4.3, Table 6) - on Qwen's commitment probes, the probe turns of transparent-compact processed on average 57,300 prompt tokens not served from cache, against 21,389 for pi-vcc, 2.7 times as many, and the count includes the lookup subagent. In my measurement of 722 agent sessions (150,902 model calls, DOI 10.5281/zenodo.22759216) more than 85% of the modeled cost was work with context, and sessions longer than 200 calls (8% of sessions) gave more than 90% of the money. In a separate run of 168 sessions, 94.4% of billed tokens were re-reading context already sent (contextburn, DOI 10.5281/zenodo.22712985). So a checkpoint that is larger but read on every later turn is where the bill sits. Could you report total tokens per session including cache reads and cache writes, not only uncached prompt tokens per probe turn, and how long a session runs after a compaction?

2. On the compaction trigger (Section 3.3) - compaction points are fixed, at a 48k-token window for real sessions and about 95k for synthetic ones. In my experiment of 36 coding-agent runs (six context-clearing policies x six repeats, DOI 10.5281/zenodo.22759217), clearing after every task cost about a third more (+33.5%) than clearing every three tasks. So the trigger point is a cost knob of its own. Could you vary the compaction trigger, not only the method?

3. On confident failures (Sections 3.1.1, 3.4, 4.1, 5.4) - the checkpoint header says retained prose is "a record of what was said, not verification of its claims", and a re-orient block asks the agent to check the repository and re-read files before editing. But probe turns run with no file, shell or edit tools, and the Limitations note that real agents can re-read files. In my guides I write that every task needs its own check the agent can run; without it "looks done" is the only signal it has. And a model sounds equally sure when it is right and when it is wrong. In my practice I also hand heavy reading to a subagent so the main conversation stays clean, which is what context_lookup does. With file and shell tools switched back on, does the re-orient block make the agent actually re-read before acting, and does the share of confident failures change?

4. On data (Section 3.3) - a replication package is on GitHub, but no session text, probe text or model reply is published. I agree with blind scoring and a replication package. In my 36-run experiment the run data is public on Hugging Face (DOI 10.57967/hf/10366), so anyone can recompute it. Could you publish per-probe scores, judge labels and per-turn token counts, even if session text stays private?

Thank you for a clear and useful paper.

Competing interests

The review cites my own technical reports and dataset on coding-agent context cost and context clearing (DOIs 10.5281/zenodo.22759216, 10.5281/zenodo.22759217, 10.5281/zenodo.22712985, 10.57967/hf/10366).

Use of Artificial Intelligence (AI)

The author declares that they used generative AI to come up with new ideas for their review.