Agent fleet operations¶
Running agents as production software means the fleet has the same operational surface as any critical service: versioning, rollout waves, monitoring, recovery, and support routing. This page is about how that is done — and what happens when something breaks.
Immutable, versioned runtimes¶
- Runtime images are built in CI and published to the container registry with immutable tags and digests.
- Manifests pin exact artifacts; a deployment is always reproducible from git.
- Component releases are coupled through a release manifest that records the exact image set and the rollout waves.
Canary-first rollouts¶
New runtimes, baseline configurations, and broker/router contracts follow the same order:
- Canary — internal/operator agents first (the platform's own agents act as the test fleet).
- Low-risk wave — a small, low-risk tenant.
- Fleet — remaining tenants.
Coupled changes deploy in dependency order (capability layer first, backward-compatible; consumers second), and each component stays backward-compatible for at least one release.
Baseline discipline¶
Shared agent instructions (persona, capabilities, fleet rules) are mounted read-only from versioned ConfigMaps. Changing them requires:
- A new immutable ConfigMap version,
- re-pointing every affected deployment,
- rolling out pods (subPath mounts do not hot-refresh),
- verifying the live content inside pods.
There is no "edit the running agent" path — a change that is not in git and rolled out is not a change.
Durable sagas with verified postconditions¶
Lifecycle operations run as typed sagas — each step has an owner (an authenticated actuator) and a postcondition that must be verified before the next step starts:
- Tenant identity & enrollment
- State provisioning (storage policy, uid)
- Model and capability access (virtual key, broker token, scopes)
- Desired-state manifest PR + rollout + verifier
- Routing and support registration
- Final state projection
If an actuator is missing, the saga stays in an explicit waiting state — there is no ad-hoc "do it manually and move on" fallback. This is what makes onboarding, capability changes, and tenant changes repeatable instead of heroic.
Observability & automation¶
| Mechanism | What it watches |
|---|---|
| Health cron jobs | Gateway readiness, silent-bot detection, model-failure counters |
| Circuit telemetry | Broker/liteLLM fallback counts, open circuits |
| Placement audits | Workloads on the right node tier, no control-plane tenants |
| Pod-restart monitor | Crash loops and wedged bridges |
| Backup staleness alerts | Restore drills and backup freshness |
| Mission-control panel | Gateways, tasks, broadcasts in one view |
| LLM tracing (Langfuse) | Per-request traces and spend |
Bridges and routers expose /health (connection state + last-event age) with
liveness probes so a wedged socket auto-restarts, and they emit an ops alert
on disconnect — a silently-dead bridge is treated as the worst failure mode.
Support routing¶
Tenant incidents travel an authenticated path: tenant outbox → dispatcher → scoped support lane, with durable ACKs. There are no unauthenticated webhooks carrying credentials, and error details never leak to tenants — agents report user-friendly outcomes and route technical detail internally.
Tenant isolation invariants (enforced, not aspirational)¶
- A tenant gateway holds only its own virtual key and broker token.
- Upstream provider credentials appear nowhere in tenant pods — automated audits scan sealed secrets and live environments and fail on drift.
- Skills and workspaces are read-only, allowlisted mounts — no tenant can read another tenant's directories or the platform root.
- NetworkPolicies restrict what a gateway can reach: the model gateway, the broker, and little else.