Why Model Routing in Agents Fails: Cost, Complexity, and Latency Are Deceptive

The Hidden Cost of Model Selection

Building an agentic system that routes between multiple LLMs seems straightforward: steer lower-complexity work toward cheaper models, escalate demanding problems to capability-heavier (and pricier) alternatives, and let specialization handle the rest. According to Hugging Face’s account of IBM Research’s findings, that intuition breaks down almost immediately in production.

IBM Research evaluated this dynamic using 417 distinct tasks from the AppWorld Test Challenge, where a CodeAct agent could invoke either Claude Sonnet 4.6 or GPT-4.1. The empirical result contradicted sticker-price expectations: Sonnet incurred $79 in total expense ($0.19 per task), while GPT-4.1 reached $155 ($0.37 per task)—a 96% cost premium despite GPT-4.1’s lower published token rates. The culprit was a factor that most routing systems ignore: prompt caching.

Agentic workflows retain substantial portions of their context between reasoning steps. When models exploit that reusable data through cache hits, their effective input costs collapse. Sonnet’s cache-read pricing advantage proved decisive—it overcame both its higher baseline token cost and its longer execution paths. A router examining only published rate cards would fail here, optimizing toward phantom economics.

When Task Difficulty Hides Until Execution

Routing heuristics typically assume difficulty is knowable upfront. A prompt arriving as “analyze this contract” appears straightforward, yet execution reveals retrieval phases, compliance validation, chained tool invocations, and iterative refinement—complexity invisible before the agent starts work. Conversely, a technical prompt may land on a smaller specialized model and finish efficiently.

According to the Hugging Face report, this binary between observable and latent difficulty creates a fundamental asymmetry: routers must commit to a model before the actual workload reveals itself. Even perfect difficulty estimation would be insufficient, because production systems balance multiple objectives simultaneously. Cost, latency, vendor specialization, reliability thresholds, and governance constraints (data residency, regulatory approval, vendor whitelisting) all compete. A task theoretically suited to one model may route elsewhere because of corporate policy or privacy mandate—and the router must accommodate this gracefully.

Infrastructure and Concurrency Reshape the Tradeoff Space

Routing is not a pure classification task. It becomes a systems-optimization problem where model selection interacts with serving infrastructure, cache state, queue depth, and real-time resource availability. Latency is not model speed alone; it reflects batch queueing, cache-hit patterns, concurrent load, and inter-request dependencies that no benchmark captures.

Why This Matters

Teams deploying agentic systems often assume that published token pricing and benchmark difficulty rankings are sufficient to guide routing decisions. IBM Research’s findings suggest otherwise: effective model selection requires instrumentation of actual workload patterns, caching infrastructure, and governance constraints, not benchmark extrapolation. Organizations building routers should treat the problem as infrastructure optimization—measuring end-to-end cost and latency under representative load—rather than as a classification task. The payoff is significant: suboptimal routing can inflate costs by 2x, while “optimal” routing based on pricing alone may actually be worse than simpler heuristics that account for cache reuse and task-invisibility.