Architecture

The platform is layered so that each concern has exactly one owner: model routing, capability access, identity, and secrets. Agents and harnesses sit on top and consume everything through narrow, well-defined interfaces.

One platform, three planes: consumers reach models through LiteLLM and capabilities through the broker; identity, secrets and data back everything.

Request flow — a chat message to an agent

1. A user sends a message to an agent gateway (Matrix, WhatsApp, or a local harness session).
2. The gateway's router classifies the message in milliseconds (regex + heuristics, no LLM in the hot path) and decides the backend: coding work, chat, or a capability like search or voice.
3. Chat/model work goes to LiteLLM with the gateway's virtual key. LiteLLM resolves the alias to a provider — possibly through the smart router, which picks the cheapest tier that answers and falls back across providers on failure.
4. Capability work (web search, transcription, TTS, a phone call, a browser session) goes to the Provider Broker with the gateway's scoped token. The broker enforces quotas, budgets, and allowlists, then calls the upstream provider. The upstream API key never leaves the broker.
5. The response returns through the same path; usage events land in the broker's audit store and LiteLLM's usage records.

Why this shape

• One place to rotate a provider: changing a key or retiring a provider is a config change in one service, not a fleet-wide secret rollout.
• Failover is exercised, not hoped for: the platform runs permanent failover drills (canary aliases that force a real provider error) so fallback chains are proven continuously.
• Tenants are isolated by token, not by network: every gateway has its own virtual key and broker token with its own scopes and budgets.

Public diagram

This is a simplified public view. The production cluster has additional control-plane services (observability, backup jobs, CI runners) that are intentionally not shown here.