Skip to content

LiteLLM proxy

LiteLLM is the model gateway of the lab: a self-hosted, OpenAI-compatible proxy that normalizes 60+ model aliases across direct APIs, subscription plans, and aggregators behind a single endpoint, a single master key, and per-consumer virtual keys.

What runs where

  • Proxy: a stateless deployment behind an internal service; the UI and the /v1 API share the same proxy. The admin UI sits behind Caddy auth on the private mesh — not exposed to the public internet.
  • Database: usage, keys, and spend records live in managed PostgreSQL.
  • Redis: rate limiting and routing state.
  • Config: the alias catalog is a versioned ConfigMap — providers, models, weights, fallbacks, and canaries are declarative, reviewed, and rolled out through the normal pipeline.

Virtual keys per consumer

Every consumer (each agent gateway, each local harness) gets its own virtual key with a name, budget, and model allowlist. Benefits:

  • Revocation is instant and scoped: kill one key, not the master.
  • Spend is attributable: the proxy records usage per key, which feeds cost telemetry per tenant and per workload.
  • Leaks are bounded: a compromised virtual key exposes only the aliases and budget granted to it — never the master key or upstream credentials.

The alias catalog

The catalog is organized into sections, each with a purpose:

Section Examples Purpose
Individual aliases anthropic/claude-sonnet-5, grok-4.3, deepseek-v4-pro, gpt-5.6-luna, minimax-m3, glm-5.2-subscription Direct or subscription-backed single models
Tier groups chat-fast, chat-balanced, chat-smart, research-deep, code-fast, code-smart The four stable tiers used by every harness (plus coding tiers)
Multi-key capacity groups one alias, multiple provider accounts with weights Scale capacity across keys/orgs
PAYG mirrors *-openrouter, smart-router-openrouter Pay-as-you-go continuity for subscription tiers
Subscription shims ChatGPT plan proxy, GLM coding plan, Kimi CLI shim, OpenCode Go OpenAI-compatible shims over personal subscription plans (token refresh, plan-specific endpoints)
Canaries canary-minimax-429, canary-glm-429 Permanent failover drills: force a real provider error, assert the mirror serves it
Vision / embeddings vision alias + direct-OpenAI mirror Special-purpose models with their own fallback credit pools

Why subscription shims are interesting

Part of the catalog is backed by personal subscription plans exposed through small OpenAI-compatible shims (ChatGPT plan proxy, Z.ai coding plan, Kimi CLI endpoint, OpenCode Go): the shim holds the plan's OAuth/token credentials, refreshes them in memory, and exposes a standard /v1/chat/completions interface to LiteLLM. This turns flat-rate plans into first-class routing targets — the same pattern a company would use to pool enterprise plan entitlements behind one gateway. ToS and breakage risks are documented per shim, and every shim has a documented fallback (traffic re-routes to the PAYG mirror).

Operational patterns

  • max_tokens parity between a tier and its fallback so a failover does not re-truncate a reply.
  • Cost metadata on aliases (input_cost_per_token, output_cost_per_token) so spend records are accurate even for plan-backed aliases (plan cost is policy, not per-token).
  • Rollout discipline: config changes are PRs; the proxy restarts quickly and the catalog diff is reviewable.
  • Readiness: the proxy answers health checks so the ingress and the fleet's router can fail fast.

Wire compatibility

Because the endpoint is OpenAI-compatible, any tool that speaks the OpenAI API — including the local coding harnesses (see Harness integration) — can be pointed at the lab gateway with two environment variables: OPENAI_BASE_URL and OPENAI_API_KEY (the virtual key).