Smart router

The Auto Router is the lab's answer to "which model should answer this?" Instead of hard-coding a model per workload, one alias — smart-router — classifies every request and routes it to the cheapest tier that can answer it well.

How routing decisions are made

Keyword rules first, a cheap LLM classifier second, a <1 ms fail-open scorer third — the router never blocks on the classifier.

1. Deterministic keyword rules first — greetings and trivial inputs ("hi", "hola", "thanks", …) go straight to SIMPLE without any model round-trip.
2. An LLM classifier (chat-fast, 1.5 s timeout) scores everything else into MEDIUM, COMPLEX, or REASONING.
3. Fail-open safety net — if the classifier errors or times out, a local <1 ms heuristic scorer takes over. The router never blocks on the classifier.
4. Session pinning — the chosen tier is pinned for the rest of the user task/tool loop via an opaque session id (30-minute pin), so a conversation doesn't flip models mid-task.

The four outputs are the stable tier aliases (chat-fast, chat-balanced, chat-smart, research tier), so the provider/fallback policy stays inside the routing plane — applications just see smart-router.

Cost discipline baked in

• The classifier itself is the cheapest model in the catalog.
• Keyword rules cost nothing (regex, no model call).
• Tier escalation is explicit: "escalate only on real failure" is the documented policy; the router makes it the default behavior.
• A smart-router-openrouter mirror re-runs the same deterministic policy with pay-as-you-go destinations — if the primary backends degrade, the policy survives on a different credit pool.

Weighted capacity groups

Tier groups support multiple provider accounts behind one alias with weights:

- model_name: chat-smart
  litellm_params:
    model: openai/glm-5.2
    api_key: os.environ/ZAI_API_KEY
    api_base: https://api.z.ai/api/coding/paas/v4/
    weight: 10

Adding a second account is a config change (a documented "capacity slot"), not an application change. This is the pattern for capacity scaling without duplicating routing logic.

Fallback chains

Beyond tier routing, individual aliases declare fallback chains so a single model outage degrades gracefully:

fallbacks:
  - kimi-k2.6: ["qwen3.7-max", "mimo-2.5-pro", "deepseek-v4-pro"]
  - kimi-k2.5: ["kimi-k2.6", "qwen3.7-max", "mimo-2.5-pro", "deepseek-v4-pro"]

And for the fleet's subscription-backed aliases, the outer agent chain owns provider order (subscription → plan mirror → PAYG mirror), with each rung explicit rather than invisible.

Failover drills — proven, not hoped

The lab runs permanent failover drills through canary aliases:

• canary-minimax-429 raises a real RateLimitError on every call — the same exception class as an actual provider 429 — and the request must be served by the OpenRouter mirror.
• canary-glm-429 does the same for the smart/research tier backend.
• The drill asserts the response header shows the mirror's API base, i.e. the fallback machinery really engaged.

This means the router's retry policy and fallback paths are exercised in production continuously — a 429 during an incident is not the first time the code path has run.

Why this matters

The smart router is a small piece of the platform that demonstrates a large skill: turning model selection into a reliable, observable, testable service. The same design ideas — deterministic fast paths, a cheap classifier, fail-open degradation, session pinning, weighted capacity, permanent failure drills — are exactly what production AI platforms need when the model catalog is a moving target and cost is a first-class constraint.