LiteLLM Integration

Date: 2026-05-10 (accepted) · 2026-05-15 (Phase 5 closed) Status: Implemented (Phase 1–5) Depends on: None. Blocks: bulletproof-hosted-agents (per-agent attribution, budget hook, core/skill split). Companion to: Observability — operator-facing reference for the trace family this plan emits.

The Decision

Stand up LiteLLM as a stateless platform-level peer service so every hosted poker agent's LLM call routes through one gateway (own contract, container, deploy lifecycle, no Postgres connection) with per-agent model selection from hosted_agents.model and optional opt-in per-agent backup from hosted_agents.fallback_model. Audit every inference into a per-agent cost ledger (agent_inference_charges). Never silently switch providers — when an agent's chosen providers exhaust, the auto-fold path fires; the gateway never picks a model the player didn't choose.

The bug class this plan structurally prevents: "one revoked Gemini key freezes every hosted-agent table simultaneously with no audit trail of what we paid the provider for the inference we never got."

Eight committed decisions

1. LiteLLM as a stateless peer container. Its own env/contracts/litellm.contract.json (env wiring superseded by the Doppler migration 2026-06-10 — the container now lives in the committed root compose with vars from Doppler; see How env vars work). Reads litellm.config.yaml from the repo root at boot. Outbound to provider APIs + Dash0 OTLP collector only. NO Postgres. NO virtual key DB. NO team budgets. The cost-ledger lives in our schema (packages/db/src/schema.ts:1306 agent_inference_charges), not LiteLLM's.

2. Per-agent model selection, no platform-level cross-provider fallback chain. Agent's hosted_agents.model column → LiteLLM alias on the per-call request body. Optional hosted_agents.fallback_model → per-call fallbacks: [string] request param. When primary + opt-in backup BOTH exhaust, hosted-agent-service's handleTurn emits a safe action (check if legal, else fold) — the gateway never silently switches to a different alias.

3. Skill-split refactor under apps/hosted-agent-service/src/. core/llm-call.ts (provider-agnostic POST to LiteLLM, returns {action, cost, model, provider, tokensIn, tokensOut, latencyMs}); core/prompt-assembly.ts (generic skill-aware composer); skills/poker/cash-nlhe/{prompt,index}.ts (first concrete skill); core/inference.ts (orchestrator). T13 grep invariant scripts/check-hosted-agent-core-boundary.sh enforces poker-concept-free core/.

4. One audit row per inference call into agent_inference_charges with full domain attribution: agentId, ownerUserId, tableId, handNumber, actionIndex, skillId, model, provider, costUsd, chargeChips, tokensIn, tokensOut, latencyMs, success, errorCode, createdAt. The failure path (provider exhausted) writes success=false + zeros so audit history is gap-free.

5. Two-axis pricing: cost_usd (COGS) and charge_chips (revenue) are different columns. cost_usd comes from LiteLLM's x-litellm-response-cost response header (what we paid). charge_chips comes from MODEL_CHARGES in apps/hosted-agent-service/src/core/model-charges.ts (what we billed). Never one column doing both. Pre-launch routing shortcut: every alias points at Gemini under the hood; cost_usd reflects Gemini's actual cost; charge_chips reflects the alias-specific flat rate.

6. Per-table inferenceTimeoutMs. tables.settings.inferenceTimeoutMs (Zod-bounded 1000–60000ms, default 15000ms) controls hosted-agent-service's AbortController timeout per LLM call. Read fresh per call via getInferenceTimeoutMs(getDb(), tableId) — DB UPDATE picks up on the very next agent inference, no redeploy. Format-tuned defaults captured in schema: cash NLHE 15s, future SnG 8s, future Turbo MTT 5s.

7. Mock LiteLLM stub at test/mocks/litellm-stub/ speaks the same /v1/chat/completions shape with configurable failure modes. Two configuration paths: ?delay_ms=N&status=N&cost_usd=N&action=...&amount=... query-param mode for direct stub calls; body.model = chaos-{503,500,401,timeout,malformed,empty,delay-{ms},success-{micros}} body-model mode for LiteLLM-routed chaos (LiteLLM forwards body.model unchanged to upstream). Reused by bulletproof-hosted-agents for deterministic chaos / cost / timing tests.

8. Native LiteLLM OTel callback emits gen_ai.* spans natively. Outbound traceparent propagation from hosted-agent-service's POST to LiteLLM is working today; incoming your_turn → hosted.turn propagation is a known gap (T10 blocked) scoped to a follow-up plan.

Architecture

Per-inference sequence

Schema boundary

Rejected alternatives

Alternative	Why rejected
Direct Gemini fetch + custom multi-key rotation in hosted-agent-service	Reinvents LiteLLM. No standardized cost computation. No battle-tested fallback orchestration. Violates "Do Not Reinvent the Wheel".
Vercel AI Gateway as the primary gateway	Vendor lock on a load-bearing path. Kept as an optional opt-in fallback provider, not primary.
Vercel AI SDK in-process (no separate gateway)	User explicitly preferred proxy pattern + cross-service inference observability chokepoint.
LiteLLM in pass-through Gemini-shape mode	Blocks multi-provider fallback because LiteLLM can't translate Gemini's wire format to other providers in pass-through.
Platform-level cross-provider fallback chain (silent Gemini → Anthropic → OpenAI)	Violates feedback_no_silent_fallbacks. The player picks the model; the gateway must not switch to a model the player did not choose.
LiteLLM's built-in Postgres-backed spend tracking	Per-API-key granularity instead of per-agent-turn. Couples gateway container to our schema. Blocks future swap to Vercel Gateway / Portkey.

Deferred verifications

• U1 — Total spend reconciliation to provider billing portals. Provider portals have delayed/non-API access to billing data; CI cannot fetch ground-truth real-time. Mitigation: weekly operator reconciliation script (see Observability — U1 Reconciliation Runbook) comparing aggregated cost_usd per provider against billing dashboards. Drift > 5% triggers a follow-up to update LiteLLM's price catalog or investigate cost-attribution bugs.
• T10 (cross-service trace) — partial. litellm.* spans correlate via outbound traceparent; incoming your_turn → hosted.turn propagation is a known gap requiring two-sided WS-traceparent injection + extraction. Scoped to a follow-up plan.
• T16 (model column semantics on fallback) — ADR-level. row.model records the agent's CONFIGURED primary alias; the ADR's original wording "alias picked OR fallback if primary failed" remains undecided. Captured as blocked trajectory state pending a model-vs-served-model amendment (Phase 6 falsification work).

Build sequence

1. Phase 1 (Foundation) — container + contract + env vars + mock stub.
2. Phase 2 (Skill split refactor) — core/ + skills/poker/cash-nlhe/, T13 + T14 grep invariants.
3. Phase 3 (Multi-provider config + per-call model selection) — litellm.config.yaml with per-provider aliases; per-call model + fallbacks from agent record.
4. Phase 4 (Auto-fold + cost capture + ledger) — agent_inference_charges table + per-call write + safe-action path on exhaustion.
5. Phase 5 (OTel polish + per-table timeout) — gen_ai.* span verification + inferenceTimeoutMs migration from hardcoded to per-table setting.
6. Phase 6 (Falsification) — capture audit-ledger durability, agent-delete FK, missing chaos coverage, silent-charge regression, T16 ADR amendment.

Bulletproof-hosted-agents Phase 1 (schema migration adding skill + model + fallback_model columns) lands in parallel with this plan's Phase 1 as the schema prerequisite for Phase 2.

Closed scope

• LiteLLM container + contract + composable.env wiring (superseded by Doppler migration 2026-06-10 — see How env vars work)
• Mock stub at test/mocks/litellm-stub/
• Skill-split refactor (T13/T14 grep invariants live)
• Per-agent model selection via per-call request body
• agent_inference_charges schema + recordInferenceCharge helper
• Auto-fold safe-action path
• LiteLLM native OTel → Jaeger / Dash0
• Per-table inferenceTimeoutMs field + read helper + agent-connection wire
• Hardcoded INFERENCE_TIMEOUT_MS constant removed
• Phase 2 back-compat shim apps/hosted-agent-service/src/inference.ts deleted

Phase 6 falsification — closed (2026-05-16)

All falsification hypotheses landed as Phase 6 fix items + verification scenarios:

Fix	Trajectory	Scenario	Summary
F1	T16	s10/s10b/s11/s11b	Direction A: served_model column added (nullable text). model = configured billing-axis primary; served_model = actually-routed ops-axis alias. NULL on full exhaustion. Migration 0041.
F2	T19	s18	recordInferenceChargeWithRetry wraps the write with 3 bounded attempts (exponential backoff). Triple-gated POST /internal/test/inference-write-fail-next primes failure for harness verification.
F3	T20	s19	agent_inference_charges.agent_id FK flipped from ON DELETE CASCADE to ON DELETE SET NULL + column nullable. Audit history survives agent deletion. Migration 0044.
F4	T25	s20	UNIQUE INDEX agent_inference_charges_turn_uq on (agent_id, hand_number, action_index). recordInferenceCharge uses .onConflictDoNothing() returning existing row's id on conflict — duplicate calls are idempotent. Migration 0043.
F5/F6/F7	T21/T22/T23	s14/s15/s16	Chaos coverage: chaos-timeout / chaos-malformed / chaos-empty all produce failure rows with the correct error_code regex.
F8	T24	s21	chargeChipsForAlias returns { chargeChips: 0, missing: true } on unpriced aliases — no silent default. Ledger row writes charge_chips=0, error_code='charge_alias_missing' (success=true). s10b + s17 retrofitted.

18 hosted-agent scenarios green end-to-end.