PREreview of Planner-as-Router: Joint Plan-Time Model Routing for Cost-Efficient Multi-Agent Workflows

## Summary

Per-call routers decide one node at a time and never see workflow structure; a cheap model that errs at step one poisons everything downstream. Planner-as-Router (PaR) moves tier selection into planning: as the planner decomposes a query into a subtask DAG, it assigns each subtask a tier — small, mid, or frontier (Haiku 4.5 at $1/$5, Sonnet 4.6 at $3/$15, Opus 4.7 at $5/$25 per MTok) — with dependencies visible before any specialist runs. The planner (Haiku 4.5, forced structured output) emits a typed plan per subtask — id, description, specialist, tier, depends_on — plus a required cost_rationale justifying every tier choice. A Haiku classifier short-circuits SIMPLE queries; a fixed dispatcher runs the DAG with a uniform retry/escalation policy (retry at tier up to 3×, escalate one tier once, null propagation) and a $75 kill switch. EntBench: 54 enterprise tasks across 7 classes, graded by execution — SQL against live PostgreSQL 15, MongoDB pipelines against live MongoDB 8.0 — over 8 routers × 3 seeds (1,157 evaluations). Results (excluding SQL-Compose, n=138): PaR 26.1% accuracy at $0.0101/task vs all-frontier 29.0% at $0.0182 — a 44% cost cut for a 2.9-point drop. Tier mix: 45.4% small / 46.4% mid / 8.2% frontier, 1.8 subtasks per plan. Accuracy gaps are not statistically separable (±6-point CIs); routers separate on cost.

## Strengths

1. **Execution-based grading is the right call.** No LLM-as-judge: correctness means the generated SQL reproduces the gold result against a live database. Bad joins, empty aggregates, and malformed plans fail here where text comparison would pass them. For routing research — where a grader's own errors contaminate the routing signal — this is the evaluation discipline the field should adopt.

2. **Load-bearing ablations, including a clean negative.** Removing the required cost_rationale field collapses Cross-Recon to 0% at over 4× PaR's cost — the rationale steers assignment, not decorates it. PaR-Lite (classifier only) matches the cheapest routers on accuracy at PaR-level cost, attributing the benefit to integrated planner reasoning over the whole workflow.

3. **Rare statistical candor.** Accuracy gaps sit inside ±6-point CIs and the authors frame the advantage as frontier position, not an accuracy win; they disclose all-small beats PaR on cost at equal accuracy; they admit their first FrugalGPT baseline never escalated and rebuilt it faithfully; the compounding section is labeled a preliminary probe, not a result.

## Major concerns

1. **The hardest compositional class scores 0% for every router.** SQL-Compose — 8 of 54 tasks, where SQL feeds a downstream reasoning consumer — is uniformly unsolved (79% of failures at the first SQL step) and excluded from the analysis. Absolute accuracies are 22–29%. An "enterprise agentic" benchmark whose most compositional class is a uniform zero is a thin foundation for production conclusions.

2. **Single provider, single planner, shallow plans.** One provider's small model assigns that provider's tiers — the mixed-provider setting is untested. Tier-assignment quality inherits planner quality, with no learned fallback when the planner misjudges difficulty. Plans average 1.8 subtasks, so compositional depth — where plan-time visibility should matter most — is barely exercised.

3. **The most important finding is the least measured.** The compounding-error factor ρ — 3.33 for PaR vs 1.27 for all-frontier — comes from a 21-task pilot on a non-shared task population, with the authors' own caveat that the baseline may be mismatched. Cascade amplification is the central risk of cheap routing on real workflows, and it arrives as a hypothesis.

## Practitioner perspective

Two things I would carry into production. First, the required cost_rationale field: forcing the planner to justify tier choices in the same call is a cheap, portable pattern — the ablation shows it is load-bearing — and it drops into any graph framework as a prompt change plus a dispatcher. Second, the honest cost accounting (retries and judge cost billed, kill switch) is how routing evaluations should report. But 44% is not a savings promise: it is measured at indistinguishable accuracy on 54 shallow-plan tasks, and the pilot ρ=3.33 hints the compounding penalty could eat those savings on deeper real workflows.

## Overall assessment

A strong, unusually honest systems paper: plan-time routing is genuinely useful, EntBench's execution-based grading raises the bar, and the authors disclose limitations rather than smoothing them over. Recommend with revisions: complete the compounding measurement on the full task set, extend to a mixed-provider setting, and test on deeper plans.

Competing interests

The author declares that they have no competing interests.

Use of Artificial Intelligence (AI)

The author declares that they used generative AI to come up with new ideas for their review.