Skip to content

LLM Gateway

Every model in the lab is reachable through one OpenAI-compatible endpoint: a central LiteLLM proxy that owns all provider credentials and exposes 60+ model aliases to every harness and agent.

LLM gateway — harnesses through LiteLLM and the smart router
Every harness, every model, one endpoint: LiteLLM normalizes 60+ aliases, the smart router picks the tier.

What the gateway buys you

  • One key, one endpoint, one wire format. A harness that speaks the OpenAI API can use any model in the lab — DeepSeek, GLM, MiniMax, Kimi, GPT, Claude, Gemini, Qwen, Grok — without changing code.
  • Provider credentials never reach consumers. Harnesses and gateways use virtual keys; the master key and upstream keys live only in the proxy.
  • Aliases are policy. A consumer asks for chat-smart; the proxy decides which provider and which account serve it, and can change that decision without touching the consumer.
  • Cost control. The cheapest tier that answers is the default; paid providers are explicit opt-ins or last-resort fallbacks; budgets are enforced per tenant.
  • Failover is continuous. Weighted multi-key tier groups, provider fallback chains, and permanent failover-drill aliases keep the routing plane proven under real failure conditions.

The alias tiers

Four stable tiers cover nearly every workload — the same four names every harness exposes:

Alias Use when Backing (current policy)
chat-fast Cheap, fast, no reasoning — mechanical edits DeepSeek V4 Flash (direct)
chat-balanced Middle ground — 1M ctx, function calling DeepSeek V4 Flash (direct)
chat-smart Default — 1M ctx, function calling GLM-5.2 (coding plan)
research-deep Deep reasoning, long turns GLM-5.2 (coding plan)
smart-router Automatic: let the router pick Auto Router v2 (see Smart router)

Coding-specific tiers (code-fast, code-smart) and special-purpose aliases (vision, embeddings, canaries) complete the catalog.

Reliability engineering

  • Permanent failover drills: canary-* aliases force a real upstream rate-limit error on every call and must be served by the mirror route — the fallback machinery is exercised in production continuously, not only during incidents.
  • Continuity routes: every subscription-backed tier has a pay-as-you-go mirror (e.g., OpenRouter) so a plan outage degrades to a slightly more expensive provider instead of a dead endpoint.
  • Weighted tier groups: multiple upstream accounts/keys behind one alias with weights for capacity scaling — with commented "capacity slots" ready to activate.

Note

Names of current backings are descriptive of the routing policy; the routing plane is designed so backings can change without consumer changes.