2026-05-15 — LiteLLM gateway Phase 1–5

Accepted 2026-05-10, Phase 5 closed 2026-05-15. Hardening Phase 6 authored, execution pending.

What changed

• Stateless LiteLLM gateway live. Every hosted-agent LLM call routes through ${LITELLM_URL}/v1/chat/completions. Container has no Postgres connection — audit/cost lives in our schema only. Reference: LiteLLM Integration ADR.
• Per-agent model selection. hosted_agents.model column controls the LiteLLM alias; hosted_agents.fallback_model enables an opt-in per-agent backup via per-call fallbacks: [string]. No platform-level cross-provider fallback chain — when both exhaust, hosted-agent-service emits a safe action.
• Per-call cost + charge ledger. Schema public.agent_inference_charges writes one row per inference: cost_usd (COGS from LiteLLM's x-litellm-response-cost header), charge_chips (revenue from per-alias MODEL_CHARGES config), provider (actually-served via api-base hostname match), full domain attribution (agentId, ownerUserId, tableId, handNumber, actionIndex, skillId, tokensIn, tokensOut, latencyMs, success, errorCode).
• Auto-fold safe-action path. Provider-exhausted calls produce success=false rows + a check/fold WS action so the table doesn't freeze. Coverage: test/poker/s11 (no-backup), s11b (primary+backup down), s13 (docker compose stop litellm).
• Skill-split refactor. apps/hosted-agent-service/src/core/ is game-agnostic; apps/hosted-agent-service/src/skills/poker/cash-nlhe/ is the first concrete skill. T13 + T14 grep invariants enforce structurally. Back-compat shim apps/hosted-agent-service/src/inference.ts deleted in Phase 5.
• Per-table inference timeout. tables.settings.inferenceTimeoutMs (Zod-bounded 1000–60000ms, default 15000ms) replaces the hardcoded constant. apps/hosted-agent-service/src/agent-connection.ts reads it fresh per call via getInferenceTimeoutMs(getDb(), tableId) — DB UPDATE picks up on the very next agent inference, no redeploy. Migration 0038_backfill_inference_timeout_ms.sql backfills existing rows. Reference: Observability — Per-Table Inference Timeout.
• Mock LiteLLM stub. Express server speaking /v1/chat/completions shape at port 4001 (test profile only). Two configuration modes: query-param (?delay_ms=N&status=N&cost_usd=N&action=...) for direct stub calls, body-model (body.model = chaos-{503,500,401,timeout,malformed,empty,delay-{ms},success-{micros}}) for LiteLLM-routed chaos. Reused across bulletproof-hosted-agents.
• LiteLLM native OTel. gen_ai.* spans visible in trace pipeline (verified against local Jaeger; production env-var swaps to Dash0). Outbound traceparent from hosted-agent-service POST to LiteLLM works; litellm.* spans parent under POST. Reference: Observability — LiteLLM Gateway Spans.

Trajectory

Row	Asserts	State
T1	Happy path: agent acts before turn timer	passing (s10)
T2	One-key-revoked: hand completes via remaining keys	passing (s12)
T3	No-backup primary-down: agent auto-folds	passing (s11)
T4	Opt-in backup recovers	passing (s10b)
T5	Primary + backup both down: auto-fold + hand continues	passing (s11b)
T6	Gateway container down: auto-fold	passing (s13)
T7	Every success → one row with full attribution	passing (s10)
T8	Failed-then-recovered fallback → single row with backup's exact cost	passing (s10b)
T9	Auto-fold row has success=false, cost_usd=0	passing (s10/s11/s11b)
T10	Trace span hierarchy poker.action → hosted.inference → gen_ai.*	blocked — cross-service your_turn propagation gap, two-sided fix scoped to follow-up
T11	LiteLLM peer in service map	passing (Jaeger verified)
T12	New model in yaml → callable by alias	passing
T13	Skill split — no poker concepts in core/	passing (grep invariant)
T14	No direct Gemini calls outside gateway	passing (grep invariant)
T15	Missing LITELLM_URL fails service at boot	passing
T16	Cost row records actually-used model on fallback	blocked — model column records configured primary; ADR amendment scoped to Phase 6
T17	/model_group_info returns model list with rates	passing
T18	Inference timeout from tables.settings.inferenceTimeoutMs (no redeploy)	passing (s17)
T19–T25	Hardening: audit-ledger durability + agent-delete FK + chaos coverage + no-silent-fallback charge + UNIQUE	planned (Phase 6 hardening work)

What's next

• Phase 6 hardening execution: F2 (DB-write retry), F3 (FK behavior fix), F4 (UNIQUE constraint), F5/F6/F7 (s14/s15/s16 chaos coverage), F8 (chargeChipsForAlias no-silent-fallback + retrofit s10–s13 charge assertions), F1 (T16 ADR amendment).
• T10 follow-up plan: two-sided WS-traceparent injection (game-server your_turn payload) + extraction (hosted-agent-service WS-message receive → hosted.turn parent context).
• Multi-key Gemini config + cooldown verification (T2 currently exercises per-call retry-fallthrough, not allowed_fails cooldown isolation — needs sustained-traffic verification once multiple Gemini keys are provisioned).