Model routers: one endpoint, many models

Hard-coding model: "gpt-4" into an application is the AI equivalent of hard-coding a database connection string in every function. It works until the model is deprecated, too slow, too expensive, rate-limited, or simply not the best choice for that particular request.

What a router does

A router sits between your application and your model providers and answers one question per request: which model should handle this?

                ┌──────────────┐
   request ───▶ │    ROUTER    │ ──▶ local / fast / cheap
                │              │ ──▶ cloud / strong / pricey
                └──────────────┘
                  ▲      ▲      ▲
             task class  cost  availability

Routing strategies, cheapest to smartest

1. Static tiers. Classify the request once (trivial / normal / complex) and map each tier to a pool. Simple, predictable, surprisingly effective — this handles 80% of the value.

2. Round-robin within a pool. Spread load across equivalent models. Also gives you load-balancing across API keys and accounts.

3. Cost-aware. Cheapest model that has historically succeeded at this task class wins.

4. Capability-aware. Detect what the task needs (vision? tools? 200k context? JSON mode?) and filter to models that support it before ranking by cost.

5. Health/latency-aware. Skip a provider that's 500ing or running slow, without the user ever noticing.

6. Learned routing. Train a small classifier on your own success/failure history. Powerful, but only worth it once you have the history.

The non-negotiable part: fallback

Routing is half the job; fallback is the other half. Every provider will fail eventually — rate limits, outages, expired keys, depleted credits. A chain like:

primary → secondary → tertiary → local model (always available)

means a cloud outage degrades quality instead of breaking the product. Putting a local model at the end of the chain is the cheapest insurance in AI engineering.

Things that bite you

• Model IDs change. Vendor deprecations land on a schedule. Centralise the IDs in config, never in application code.
• Token accounting drifts. Different models count tokens differently; if you bill or budget, measure per model, not globally.
• Silent quality regressions. A router that quietly falls back to a weaker model is indistinguishable from a bug to your users. Log which model actually served every request.
• Config rot. A malformed router config often fails open — silently routing to a default — rather than erroring. Validate your config at boot.

What good looks like

Every request carries: the classified tier, the model that was chosen, why, and the fallback chain position if any. Then you can answer the only questions that matter — what did this cost, what actually served it, and what happens when a provider disappears tomorrow.

Our own platform runs exactly this: tier pools, a classifier in front, health-based skipping, and a local model at the end of every chain.