Cheapest per token is not cheapest per task

Findings

1. The pre-registered hypothesis held. Letting the parent session name the worker model while writing the handoff brief cut cost per completed task by 76% (95% CI 70% to 80%) versus spawning every worker on Opus, with no loss in completion (100% vs 99%). Billing the router’s entire turn still leaves a 50% saving.
2. But a fixed default did better. Always spawning on Sonnet completed 100% of tasks at $0.099 each, 23% below the router. The router chose Sonnet 87% of the time, and every departure from Sonnet added cost. The saving came from not defaulting to the most expensive model, not from choosing per task.
3. The cheapest model was not the cheapest per task. Haiku completed 84% of tasks at $0.239 per completed task, 2.4× Sonnet. It took 3.5× the turns and re-read 7× as much cached context, so it cost more than Sonnet even on the easy tasks it passed.
4. There was almost nothing left to route. A perfect hindsight oracle would have saved only 1.3% over always-Sonnet. No task in the set required Opus. Routing can only earn money when some tasks need the expensive model and others don’t; this workload had one clear winner.
5. The literature agrees once you separate settings. Headline router savings of 45% to 98% come from single-turn chat and QA benchmarks with a 20× or wider price gap. Independent re-evaluations find routers often fail to beat simple baselines, and the newer agentic studies find routing harder: prompt caches are per model, weaker models take more turns, and task difficulty is often invisible until the agent has explored the code.

Why routing looks like free money

The routing pitch is arithmetic on a price list. If a frontier model costs 5× a small one and 70% of requests are easy, sending those to the small model should cut the bill by more than half. Early academic results supported this: RouteLLM reported cost reductions of over 85% on MT-Bench while keeping 95% of GPT-4’s quality, FrugalGPT reported up to 98%, and AutoMix over 50%. Commercial products followed. Agentic work breaks three assumptions behind that arithmetic.

1. A task is many calls, and the count depends on the model. A coding agent reads files, runs tests, edits and re-runs. A weaker model that takes three times the turns can cost more in total even at half the per-token price.
2. Each call re-reads the conversation so far. Prompt caching makes this cheaper, but caches are per model. Switching models recomputes the entire request.
3. Difficulty is often invisible up front. A one-line bug report can hide a one-line fix or a multi-module refactor. A router reading only the prompt cannot tell which.

The natural place to route in an agent system is therefore the handoff: the moment a parent session spawns a fresh worker (a subagent, a task chip, a background job) with a written brief. The worker starts with an empty cache whatever model it runs on, the parent has just read the relevant context, and the brief is the worker’s whole input. Asking the parent to also name the model costs almost nothing. This study tests whether that cheap decision pays.

How it was tested

The primary hypothesis was fixed before the confirmatory run, along with the metric, the non-inferiority margin, the policies, the trial count, the randomization seed and a hash of the task set: spawning with the model the parent recommends while it writes the brief has a lower cost per completed task than spawning with the parent’s own model, with a pass rate no more than 10 percentage points worse. The report tool marks a run confirmatory only when every registered field matches. Everything else, including the comparison against always-Sonnet, is labeled exploratory.

The 65 tasks live in three purpose-built Python repositories (invoicing, log processing, a notes CLI): 29 bug fixes, 13 features, 5 config, 5 performance, 4 migrations and a handful each of docs, refactor and tests. A calibration run beforehand measured difficulty instead of guessing it: 50 tasks were easy (Haiku passed every trial) and 15 medium (Haiku passed some). No task was passed only by Opus.

A cell passes only when the hidden tests pass, the visible tests still pass, and no file outside the task’s allowed paths changed. Grading is deterministic; there is no LLM judge. All figures are list API prices: Haiku 4.5 at $1 / $5 per million input / output tokens, Sonnet 5 at $2 / $10, Opus 5 at $5 / $25, with Claude Code writing its prompt cache at the 1-hour rate.

Results

Against the Opus default, parent-picked routing saved 76% per completed task (95% CI 70% to 80%, p < 0.001) and changed the pass rate by +0.5 points, comfortably inside the 10-point margin. The router recovered 93% of the oracle’s saving over Opus. Billing the parent’s entire brief-writing turn to routing, which overstates the cost because B also needs a brief, cuts the saving to 50%.

The comparison readers will ask about was not pre-registered, so it is exploratory: against always-Sonnet, the router cost 30% more per completed task (95% CI +9% to +59%) at the same 100% completion. Put the other way, always-Sonnet was 23% cheaper than the router.

The parent made sensible-looking choices: it sent the simplest tasks to Haiku and a few it judged risky to Opus. Both kinds of departure raised cost. Haiku’s “easy” wins were no cheaper than Sonnet’s, and the Opus picks bought insurance the tasks did not need. Even on easy tasks, where Haiku passed 95% of the time, it cost twice as much per completed task as Sonnet. On medium tasks it cost five times as much, because it failed half the time and burned its full turn budget doing so.

Why the cheapest model cost the most per task

The per-session token records explain the inversion. Almost none of the cost is fresh input. It is cache reads (the conversation re-read on every turn), cache writes (new context stored at the 1-hour rate), and output.

Two effects compound. First, turn count is superlinear in cost. If each turn adds roughly the same amount of context, total re-read tokens grow with the square of the turn count, so 3.5× the turns gives about 7× the reads. A high cache hit rate does not rescue this; Haiku’s was the highest of the three at 98%. Caching makes each re-read cheap, not free. Second, the per-token price gap is small next to the per-task behavior gap.

This also explains why a cascade (start on Haiku, escalate on failure) could not be tested properly. Its deployable checker accepted Haiku’s partially correct work, so it never escalated. When escalation was forced with the hidden tests as checker, a second problem appeared: Sonnet, inheriting Haiku’s partial edits, failed a task it passes reliably from a clean checkout. A wrong start anchors the next model.

Why routing helped less than expected

The hindsight oracle beat always-Sonnet by only 1.3% ($0.0976 versus $0.0989), and a router cannot save more than the oracle. On this workload, routing’s entire upside was realized by picking one good default, and any router with imperfect judgment could only add cost. Four conditions produced that.

1. No task needed the expensive model. Where Haiku failed, Sonnet almost always passed; nothing required Opus.
2. The cheap tier was not cheap per task. Haiku 4.5 was not cheaper than Sonnet on easy work, even when it passed.
3. The price ladder is narrow. RouteLLM’s 85% came from routing between GPT-4 Turbo and Mixtral 8x7B, a per-token gap of more than 20×. Claude’s current ladder spans 5× from Haiku to Opus, and only 2× between adjacent tiers. Differences in tokens per task easily swamp gaps that small.
4. The harness had already captured the easy wins. Claude Code already uses Haiku for background work and exploration subagents, lets parents set subagent models, and offers effort levels. In the author’s own transcripts, parents routed subagents to Sonnet 9 times in 10.

What the wider research shows

• Headline results come from single-turn benchmarks. FrugalGPT, Hybrid LLM, AutoMix, RouteLLM and MixLLM measure savings per query against a much more expensive model, with a lenient quality bar (for example 95% of GPT-4) and a price gap of 20× or more.
• Independent re-evaluations are less kind. LLMRouterBench (400,000+ instances, 33 models, 10 routers) finds several recent approaches, including commercial routers, fail to reliably outperform a simple baseline. RouterArena finds no router leading on every metric. Other work shows routers collapse to the strongest model under generous budgets, and can be pushed there by short appended text.
• Agentic studies find routing harder. Swapping models mid-trajectory changes most later actions, so offline router evaluation on agent logs is unreliable. Prompt-only routing on software tasks has an error floor because difficulty is hidden in the repository. In one scout-then-fix study the saving held with the router removed, so it came from the handoff design, not the routing decision.
• Commercial claims are mostly unmeasured. The only independent multi-vendor measurements found use single-turn QA. No independent study of real-world savings from Copilot Auto, Cursor Auto, Windsurf or OpenRouter Auto on agentic coding was found, and none of these tools publish cost per completed task.

Choosing among models once per task can pay when tiers genuinely differ in which tasks they can solve. Switching models within a trajectory is hard to evaluate and hard to profit from. Much of the reported gain comes from how the work is split and handed off rather than from the routing classifier. The PDF has the full comparison, with sources.

When routing pays and when it does not

• Tends to pay when the per-task (not per-token) price gap between tiers is wide; a large share of traffic is clearly easy and recognizable from the request; some work genuinely needs the expensive model, so the oracle beats every fixed default by a real margin; requests start fresh; and a reliable verifier exists for cascades.
• Tends not to pay when it switches models inside a long, cache-warm conversation; the cheap model takes more turns or fails and retries; one model is already cheapest per completed task; the harness already routes the easy wins; routers are judged against the most expensive model instead of the best fixed default; or difficulty is only visible after exploration.

Limitations

• Synthetic, well-specified tasks. Three small Python repositories with short, clearly scoped briefs. Harder or vaguer work may open headroom or remove it. The results speak to agentic coding handoffs, not chat or knowledge work.
• No Opus-only tasks, so the study cannot show the case where routing to a frontier model is worth its price.
• Generation mismatch. Haiku 4.5 is a generation older than Sonnet 5 and Opus 5; part of its penalty may be generational. Results describe the models as served on 24 and 25 September 2026.
• Effort and brief quality held constant. Opus ran at default effort in the static arms, and every worker got the same canned brief.
• Single vendor, single operator, list prices. Price ratios differ across vendors, so the Sonnet-versus-router gap in particular may not carry over.
• The key practical comparison was exploratory. The always-Sonnet result was not pre-registered and should be confirmed in a registered follow-up.

Reproducibility: the confirmatory run was registered on 24 September 2026 against a hashed task set and config; the statistics use task-clustered bootstrap intervals, two-sided task-clustered permutation tests and a one-sided non-inferiority test, with Holm adjustment across secondary hypotheses. The run cost $202.33 at list price, plus $109.60 for calibration. The harness, raw data and the per-task pass/fail matrix for all 780 cells are in the repository.