Avalilação PREreview de RSI-Router: Evolving Subtask-Level LLM Routing and Skills for Cost-Efficient Agents

## Summary

Task-level routing misses the obvious: an agent's long-horizon task has stages, and a small model that cannot solve the whole task can still handle some stages. RSI-Router routes at the subtask level and jointly evolves model-specific execution skills, via a four-stage loop — Subtask Mining, Routing Strategy Evolution, Model-Specific Skill Evolution, Pareto-Optimal Router Selection — run for 6 iterations with 4 candidate routers each, driven by a large-model proposal agent (Codex). At inference, the small model (Qwen3.5-9B) identifies the upcoming subtask each step, picks the assigned model (itself or DeepSeek-V4.1-Flash), and appends the matching skill text to the input. Across five agentic benchmarks (ALFWorld, ScienceWorld, WebShop, SWE-bench Verified, Terminal-Bench 2.0), Table 1 reports matching or beating the DeepSeek-only baseline at 8.1–82.2% lower inference cost (51.7% average), with the Pareto frontier extended beyond 9 routing baselines. Ablations show skills reused across iterations (22 of 48 skill sources on ScienceWorld carry over) and, unusually honestly, that masking skills improves performance on SWE-bench Verified (+9.7) and Terminal-Bench 2.0 (+10.5).

## Strengths

1. **Subtask granularity is the right level for agentic routing, and the paper proves it.** The diagnosis of task-level baselines' "routing collapse" — whole-task assignment degenerating to all-small or all-large with no intermediate trade-off — is insightful and well-evidenced in Figure 3. Step-level routing without task structure is blind; subtasks are the missing middle.

2. **Jointly evolving assignments and skills is a real conceptual contribution.** Skills change what the small model can reliably do, so assignments must be revisited as skills improve — optimizing them in isolation would miss this. The cross-iteration skill reuse (Figure 5) is genuine evidence of accumulated, compounding experience rather than per-iteration overfitting.

3. **Admirably honest ablations.** Direct difficulty-based routing by the small model loses to step-level random routing; longer interaction histories don't help; and the skill-masking result on the two hardest benchmarks is reported straight (Figure 6c). Inference costs include routing calls and skill context tokens — the serving-time accounting is done right.

## Major concerns

1. **The headline savings rest on an imputed price ratio.** Qwen3.5-9B has no public API price, so its cost is estimated from GPU-hour measurements via a fixed 1:30 ratio (measured 35.96, rounded; Appendix C). That compares DeepSeek's API price against imputed self-host cost — apples to oranges, with serving ops, batching efficiency, and idle capacity absent. There is no sensitivity analysis: at 1:10 or 1:50 the "82.2%" and "51.7%" figures move substantially, and the paper never says by how much.

2. **The evolution cost is off the books.** Reported "cost" is test-time inference only. Building the router took 6 iterations × 4 candidates, each evaluated on training and validation sets, with Codex driving subtask mining, strategy generation, and skill evolution throughout. For a paper whose headline is cost efficiency, omitting the build cost while counting every serving token is a conspicuous asymmetry — and unlike SaveRouter-style analyses, no break-even or amortization volume is computed. There is also no limitations section at all.

3. **The skill machinery is net-negative where it matters most.** Figure 6c: masking execution skills improves performance by 9.7 points on SWE-bench Verified and 10.5 on Terminal-Bench 2.0 (at higher cost). The paper frames this as trade-off shifting, but the practitioner reading is blunter: on the two hardest, most realistic benchmarks — code repair and terminal ops — the evolved skills hurt. Add tiny test sets (25–64 tasks per split; Terminal-Bench's 40.00%→46.67% is ~2 tasks), a single evaluated model pair, and no distribution-shift or pool-churn test, and the generality claim is thinner than the Pareto plots suggest.

## Practitioner perspective

Two questions decide whether I would run this. First, amortization: what did the 6-iteration evolution cost in dollars, and over how many production tasks does it pay back? Second, operations: who maintains the subtask definitions and the skill library when the task distribution drifts — and skills are prompt text appended at every step, which means per-step context bloat and a prompt-injection surface that grows with the library. The routing idea (subtask-level assignment) is the durable contribution; I would adopt that with hand-written subtask policies long before adopting the evolution machinery.

## Overall assessment

A strong paper with a genuinely good routing insight, honest ablations, and correct serving-time accounting — but the cost story needs the missing halves: price-ratio sensitivity, the evolution build cost with a break-even analysis, and an account of why skills hurt on the hardest benchmarks. Recommend with revisions.

Competing interests

The author declares that they have no competing interests.

Use of Artificial Intelligence (AI)

The author declares that they used generative AI to come up with new ideas for their review.