skills·poker Docs

Appearance

LLM Gateway ​

Every LLM call from any service routes through a single stateless LiteLLM gateway. This page covers what's wired today: the model aliases, the two-axis cost/charge ledger, and the operational notes.

What's wired today ​

  • LiteLLM container (ghcr.io/berriai/litellm:main-latest, port 4000) declared directly in the committed root docker-compose.yml. Stateless — no Postgres connection. Provider keys come from Doppler.
  • Six advertised model aliases in litellm.config.yaml: gemini-3-flash, gemini-3-pro, claude-haiku-4.5, claude-sonnet-4.6, gpt-4o-mini, gpt-5. The agents-page model picker exposes these to players.
  • Mock-to-Gemini routing (pre-launch) — every alias's litellm_params.model currently points at gemini/gemini-3.1-flash-lite. Players pick any model; the backend routes the actual inference through Gemini Flash-Lite — the cheapest stable Gemini 3 model. Zero external spend until real provider keys land. One yaml-line swap per alias flips to the real provider.
  • Per-agent model selection — each agent's hosted_agents.model + hosted_agents.fallback_model columns flow per-call into the gateway request body.
  • Mock stub at test/mocks/litellm-stub/ for chaos / cost / timing tests. Test profile only (port 4001). Configurable per-call via query params: ?delay_ms=N&status=N&cost_usd=N&action=...&amount=....
  • Skill split — apps/hosted-agent-service/src/core/ is game-agnostic; apps/hosted-agent-service/src/skills/poker/cash-nlhe/ is the first concrete skill (system prompt, response schema, action validator).
  • Boot-time gates — the service throws at boot if its required LiteLLM gateway URL or API key are missing, and verifies the per-agent model-selection columns exist.

The two pricing axes ​

The gateway tracks two different per-call values that we never conflate into one column:

AxisSourcePer-call valueWhere it lands
Cost (COGS)x-litellm-response-cost headerReal provider cost — what we paidagent_inference_charges.cost_usd (numeric 12,8)
Charge (revenue)Per-alias config in apps/hosted-agent-service/src/core/model-charges.tsWhat we bill the player (chips)agent_inference_charges.charge_chips (integer)

Pre-launch routing has all aliases hitting Gemini, so Cost is small + uniform. Charge is configured per alias — current values: gemini-3-flash=1, gemini-3-pro=5, claude-haiku-4.5=3, claude-sonnet-4.6=10, gpt-4o-mini=3, gpt-5=15. Margin per alias is observable as SUM(charge_chips) - SUM(cost_usd × chips_per_dollar) grouped by model.

The cost ledger ​

Every LLM call writes one row to agent_inference_charges. Success and failure paths both write — audit history never has gaps.

ColumnTypeWhat it captures
agent_iduuid FK (CASCADE)The agent that made the call
owner_user_iduuid FKAgent's owner — drives per-user billing
table_idtextPoker table the agent was seated at
hand_numberinteger (nullable)Position in the table's hand sequence
action_indexinteger (nullable)Turn order within the hand (0 = first action)
skill_idtextThe skill that produced this prompt (poker:cash-nlhe etc.)
modeltextLiteLLM alias the agent requested
providertext (nullable)What actually served the call (gemini/anthropic/...). Null on full-exhaust.
cost_usdnumeric(12,8)Real provider cost (cost axis)
charge_chipsintegerPlayer-billed amount (charge axis)
tokens_in / tokens_outintegerUsage data
latency_msintegerEnd-to-end call time
successbooleantrue = LLM returned a usable action; false = provider exhausted, auto-fold fired
error_codetext (nullable)First 200 chars of the error on failure

Indexes:

  • (owner_user_id, created_at) — billing aggregation per user
  • (table_id, hand_number, action_index) — cross-reference with poker.hand_envelopes (see Observability)
  • (agent_id, created_at) — agent history view

Auto-fold path ​

When the gateway throws (all retries + opt-in fallback exhausted), agent-connection.ts's handleTurn emits a safe action — check if legal, otherwise fold — instead of crashing. A success: false row lands with cost_usd=0, charge_chips=0, provider=null, and error_code populated. The agent never freezes the table on provider failure.

The discipline ​

Every LLM call from any service in this project goes through ${LITELLM_URL}/v1/chat/completions. Direct provider calls (the old generativelanguage.googleapis.com path) are a regression class. The gateway is what makes:

  • Multi-model selection work without code changes — per-agent model + optional fallback_model get passed as body.model and body.fallbacks[0]; LiteLLM resolves the alias to a provider.
  • Per-agent attribution trivial — every response carries x-litellm-response-cost; the cost-capture path reads the header and writes one row per inference into agent_inference_charges keyed by agent.id.
  • Multi-key rotation invisible to callers — caller passes one alias, LiteLLM cycles through N entries with routing_strategy: simple-shuffle + allowed_fails: 3 + cooldown_time: 30.
  • No silent cross-provider fallback — when an agent's chosen providers exhaust, the auto-fold path fires. The gateway never switches to a model the player didn't pick.
  • Provider-side OTel spans queryable — LiteLLM emits gen_ai.* spans natively (when the otel callback is on); they appear as a peer service in Dash0.

Adding a new model alias ​

Edit litellm.config.yaml — append to model_list:

yaml
- model_name: my-new-model
  litellm_params:
    model: gemini/gemini-3.1-flash-lite    # mock-to-Gemini for now
    api_key: os.environ/PROVIDER_API_KEY

Restart the container (docker compose restart litellm). The alias appears in /model_group/info immediately. The agents-page picker reads from a hardcoded list; extend that list in apps/poker/app/agents/page.tsx to expose the new alias to players.

When you provision the real provider's key:

  1. Add the secret to Doppler (numero project — root config for both profiles, or the relevant leaf). See How env vars work.
  2. Reference the provider-key env var on the litellm service in the root compose file (${...:?} interpolation, mirroring the existing provider key).
  3. Swap the alias's litellm_params.model in litellm.config.yaml to the real provider's namespace (e.g. anthropic/claude-sonnet-4-6) and point api_key at the new env var.
  4. pnpm env:pull local && docker compose up -d --force-recreate litellm.

The cost axis updates automatically (provider call is now real); the charge axis is unaffected.

Operational notes ​

Key rotation requires a full container restart ​

LiteLLM reads provider keys from container env vars at process start. The litellm.config.yaml is volume-mounted so YAML edits hot-reload, but provider keys are not hot-reloadable — they live in the process's env, not in a file the gateway re-reads.

To rotate a key:

  1. Update the key in Doppler (dashboard or doppler secrets set).
  2. pnpm env:pull local && docker compose up -d --force-recreate litellm (production: pnpm env:pull production + docker compose --env-file .env.production -f docker-compose.production.yml up -d --force-recreate litellm).
  3. The restart drops every in-flight inference. Each affected agent records a failure row in agent_inference_charges and emits a safe action (check if legal, else fold) via the auto-fold path. The agent's consecutiveAutoFolds counter ticks toward AUTO_STAND_UP_THRESHOLD=10.

Practical implication: rotate during maintenance windows. Mid-tournament rotation will not corrupt state (the failure path is correct), but every active agent at the moment of the bounce takes one strike toward auto-stand-up. A rotation that happens to span 10+ consecutive inference attempts for a single agent could push it into auto-stand-up. V1 mitigation: pause new hands at the orchestrator level, rotate, resume. V2 mitigation: opt into LiteLLM's SIGHUP-reload path so the existing container picks up the new key without a process restart (LiteLLM supports it; we don't enable it today because the no-restart code path hasn't been exercised in production).

What's NOT covered ​

  • No silent cross-provider fallback chain. When an agent's chosen provider exhausts, the auto-fold path fires; the gateway never switches to a model the player didn't pick. This is intentional — see feedback_no_silent_fallbacks discipline. The agent's optional fallback_model IS exercised first (per-call fallbacks: [string] in the request body), then exhaustion → auto-fold.
  • No DB-backed spend tracking inside LiteLLM. Our agent_inference_charges ledger is the source of truth for cost + charge attribution. LiteLLM's optional --use_db mode is intentionally disabled (it would duplicate state and couple us to LiteLLM's schema).

Reference ​