Capabilities¶

Every capability the broker exposes follows the same shape: an agent calls a broker endpoint with its scoped token, the broker enforces quota and policy, calls the upstream provider with the real credential, and returns a normalized result. The agent never sees the credential.

Web search — the provider ladder¶

Raw web search is a chain: SearXNG (self-hosted, free) → Serper → Brave → Tavily, skipping rungs that are unconfigured or circuit-open. The caller can force an order or subset; the response reports which rungs were attempted before the winner, so failures are visible, not silent.

{
  "q": "current energy tariffs bogota",
  "count": 5,
  "country": "co",
  "lang": "es",
  "providers": ["searxng", "serper"]
}

{
  "provider": "serper",
  "query": "...",
  "results": [{"title": "", "url": "", "description": "", "age": null}],
  "attempted": ["searxng"]
}

The self-hosted SearXNG rung costs nothing and keeps common queries off paid APIs — a concrete example of the lab's cost discipline.

Research with citations — Perplexity¶

Synthesized answers with citations (model strictly sonar / sonar-pro), billed against a per-tenant unit budget (completion tokens):

{
  "q": "what are the best practices for k8s secret rotation",
  "model": "sonar",
  "max_tokens": 1024
}

Scraping — Crawl4AI → Firecrawl¶

Page-to-markdown conversion, chained crawl4ai (self-hosted) → firecrawl, with an SSRF-guarded URL policy (public addresses only, no embedded credentials, redirect re-check) and a 2 MiB content cap:

{"url": "https://example.com/docs", "format": "markdown", "timeout_seconds": 60}

Speech-to-text — Deepgram → OpenAI¶

Voice notes are transcribed with Deepgram first and OpenAI Whisper as fallback. The broker normalizes decodable audio (16 kHz mono WAV via ffmpeg), probes metadata, and returns specific errors rather than a generic failure:

Error	Meaning
`empty_or_silent_audio`	Empty, zero-duration, or decoded-silent audio
`empty_transcript_provider`	Provider returned HTTP 2xx but no transcript
`provider_timeout`	Upstream STT provider timed out
`budget_exceeded`	Broker quota denied the request
`all_providers_failed`	No provider produced a usable transcript

A request ID is threaded through gateway → broker → provider so a voice note can be correlated end-to-end.

Text-to-speech¶

TTS returns base64 audio with configurable voice, language, speed, and format:

{
  "text": "Hello, this is a reminder from your assistant.",
  "voice": "carina",
  "language": "en",
  "speed": 1.0,
  "format": "mp3"
}

Voice calls — Telnyx call control¶

The broker dials phone calls through Telnyx and plays a TTS message:

{
  "to": "<E.164 destination>",
  "message": "Reminder from your assistant",
  "voice": "female",
  "language": "en-US",
  "hangup_after": 60
}

A durable call row is created before dialing; the initial response only means "accepted for dialing".
Delivery is confirmed by Telnyx Call Control webhooks, verified with Ed25519 signatures and idempotent by event id.
Anti-spam policy is enforced per call: quiet hours, max calls per day, minimum interval, and importance/preferences from tenant settings.
Status and metrics endpoints track states from call_queued through call_completed, no_answer, busy, and failed.

Browser automation¶

Two modes:

Browser Use (managed): agent sessions, raw browser/CDP sessions, profiles, workspaces, files, and billing readback. Ownership is enforced server-side — resources are registered per (tenant, agent, user) and lists are filtered to owned resources. Session cost and raw-browser timeout caps are enforced from the token.
Self-hosted browser service: a persistent human-watchable browser for logins, captchas, and payments. The broker routes to self-hosted by policy and proxies tasks, handoffs, and payment-intent state (payment authorization stays human-only).

The broker rejects card data outright: a 13–19 digit run is blocked only when it passes the Luhn checksum, so bill references and NIU numbers still work, and CVV mentions are blocked unconditionally.

Image generation — Higgsfield¶

/v1/media/higgsfield/generate runs the pinned Higgsfield CLI with broker-owned credentials. If the CLI or its credentials are missing, the capability reports as degraded (HTTP 200, listed in degraded_capabilities) rather than taking down the whole broker — optional capabilities must never remove core routing from the Service.

MCP proxy — brokered services¶

Registered internal MCP servers are exposed as /v1/mcp/<service> (streamable HTTP), so gateways can configure MCP servers that point at the broker instead of the raw service:

Services must be explicitly registered with capability, kind, allowed methods, byte caps, and timeout.
Service hosts must match the broker's allowlist.
Authorization/Cookie/provider-key-style headers are never forwarded; optional upstream auth is injected from broker env.
Requests and responses are capped and audited.

Example registration shape:

{
  "services": {
    "scrapling": {
      "url": "http://internal-mcp:8000/mcp",
      "capability": "mcp:scrapling",
      "provider": "scrapling-mcp",
      "kind": "mcp-streamable-http",
      "allowed_methods": ["POST", "GET"],
      "max_body_bytes": 8388608,
      "max_response_bytes": 16777216,
      "timeout_seconds": 120
    }
  }
}

Integrations — Composio (least privilege)¶

One fixed, reviewed read operation (repository listing) is exposed through Composio with an HMAC-derived identity layer and a reservation ledger that limits attempts per minute/day/total. It is the pattern for "how to expose an external agent platform to tenants without opening a generic proxy".

Circuit breaker & introspection¶

The broker tracks upstream failures per provider/capability and opens circuits so a dead rung is skipped fast. /v1/capabilities, /v1/services, /v1/limits/status, and /v1/circuit/status let operators and agents introspect the live state — with the detail that core_errors produce HTTP 503 while degraded_capabilities do not.