skills·poker Docs

Appearance

Bulletproof Hosted Agents ​

Status

Plan accepted 2026-05-10. Phases 1–9 closed; Phase 10 cash dogfood retrospective 2026-05-19; Phase 11 SnG dogfood informally satisfied by tournament smokes.

End-to-end discipline for the hosted-agent path — create agent → seat at table → play hands → stand up → withdraw — with no silent failures, every state durable across restarts, and structural separation between game-agnostic infrastructure and format-specific skills enforced by grep invariants.

Y-Statement ​

In the context of the hosted-agent product surface where users create AI agents with strategy prompts and seat them at poker tables, facing a hard-broken path since the chain-of-custody-4 cutover (JWT-broken CRUD, orphaned seat UI, pre-COC4 WS shape, zero auth on PATCH/DELETE, in-memory-only connections that vanished on restart, and three years of carry-forward retro items), we decided to systematically rebuild the path across nine phases — CRUD auth + schema, seat-flow rewrite, per-agent attribution, dual-path leave, restart-safe persistence, multi-format wire-shape lift, attribution carryovers, structural test coverage, and audit-ledger durability — to restore a path real users can actually play with and the system handles every failure mode auditably, accepting that the rebuild crosses three services (game-server, hosted-agent-service, admin-server) and requires bulletproof-state-machine's discipline at every UI boundary.

Decisions ​

D1. Agent CRUD endpoints are JWT-and-ownership-checked ​

POST/PATCH/DELETE /hosted-agents on game-server requires Authorization: Bearer <jwt>. PATCH/DELETE additionally enforce agent.ownerUserId === identity.userId. Cross-user mutation returns 403 forbidden_not_owner. Schema-extended with hosted_agents.skill (default poker:cash-nlhe), model (default gemini-3-flash post-migration 0046), nullable fallback_model. Identity flows from auth-server JWT through the entire mutation surface — no path reads ownership from the request body.

UI consumes via useHostedAgents hook at apps/poker/src/poker/hooks/useHostedAgents.ts, threading useGameToken() into every fetch. Each mutation wraps in a ui.agent.{create,update,delete}.submit span carrying agent.id + otel.category=ui.action. Server-side mirrors with hosted_agents.{create,update,delete} spans carrying owner.userId + agent.skill + error.code on failure. Both ends register in telemetry-contract.ts; the typecheck tripwire enforces.

D2. Seat-flow is owner-permit-then-WS, in that order ​

The seat flow has three steps, run synchronously from the UI:

  1. Owner signs ERC-2612 permit via usePermitSign (owner's wallet, not the agent's).
  2. POST admin-server /api/deposit with {tableId, seatIndex, amount, permit, signerAddress}. Admin-server submits registerAndDeposit on-chain, waits for confirmation, calls game-server /internal/credit-chips which credits CTN equity at the seat.
  3. POST hosted-agent-service /seat with the agent's deployment record. Service opens the agent's WebSocket, sends sit_down { seatIndex } (no buyIn — game-server rejects pre-COC4 shape with sit_down_shape_changed).

The /seat endpoint does not handle funding. This is the same shape that human-player sit-downs use. /deploy and /undeploy are 410 Gone shims with structured endpoint_renamed responses pointing at /seat and /unseat — old clients see the rename, don't see a 404.

Affordance gates on <DashboardAgentSeat> in precedence order (wallet → agents → openSeats): missing wallet → Connect Wallet copy; zero agents → /agents link; full table → "Table Full"; otherwise seat picker. Each gate's absent-button follows bulletproof-state-machine discipline — not disabled, actually absent with an actionable alternative.

Known race (Phase 10) ​

The documented /api/deposit → /seat order works for the browser because the network stack interleaves the calls, leaving a window where /seat lands first on game-server and places the reserve before credit-chips fires. Script-side it doesn't work — await makes the race deterministic in the wrong direction (credit-chips fires before any reserve exists, returns 404 no_seat). Captured for repair — either admin-server short-polls for reserve before falling through to reconciler, or the documented order flips to /seat first.

D3. Per-agent attribution flows through every inference ​

Every agent_inference_charges row carries agentId, ownerUserId, skillId, plus tableId, handNumber, actionIndex, model, provider, costUsd, chargeChips, tokensIn, tokensOut, latencyMs, success, errorCode. The per-agent budget hook (assertAgentHasBudget(agentId) at apps/hosted-agent-service/src/core/budget-hook.ts) is called before every inference; currently a stub returning true — a future billing plan replaces it with real chip-balance / USD-budget logic. agent.budget.checked span event fires on every call.

D4. Leave is split: user-initiated vs system-initiated ​

User cash-out:

  1. UI calls hosted-agent-service POST /unseat { tableId, seatIndex }
  2. AgentConnection.standUpAndDisconnect() sends stand_up over the agent's WS (game-server marks sitting_out, broadcasts stood_up)
  3. UI POSTs admin-server /api/withdraw with the owner's JWT (linkage check applies — owner must have wallet linked to receive funds)

System auto-removal:

  1. AgentConnection tracks consecutiveAutoFolds (resets on success, increments on inference failure)
  2. When it crosses AUTO_STAND_UP_THRESHOLD=10, triggerForcedWithdraw() fires (exactly once per connection lifetime, guarded by autoStandUpFired flag)
  3. POSTs admin-server /api/withdraw { forced: true, userId, tableId } with x-service-key auth (bypasses linkage check)
  4. Admin-server routes through resolveForcedWithdrawDestination(userId) — returns last-permit-signer wallet today (explicit swap-point for the future custody-layer plan)

agent.auto_stand_up_triggered span event + forced_withdraw.resolution span gate the telemetry. The WS close handler unwinds after admin-server's settle-withdrawal broadcasts player_left.

D5. Restart-safe persistence via hosted_agent_seats ​

Migration 0042 adds hosted_agent_seats (one row per active agent+table, state='active' | 'unseating' | 'stale', last_seen_at timestamp). /seat inserts; /unseat marks unseating then deletes after 1s WS-close beat; auto-stand-up flips through the same path via settle-withdrawal.

A 30s heartbeat in apps/hosted-agent-service/src/seat-reconciler.ts bumps last_seen_at for every in-memory connection. Boot reconciler runs once at startup BEFORE the HTTP server listens — scans state='active' rows and re-establishes WS connections at ~10/sec ramp via AgentConnection.connect(). Stale reconciler (60s tick) finds state='active' rows whose last_seen_at > 5min; tries re-establish first; if that fails, force-withdraws via the same resolveForcedWithdrawDestination seam and marks state='stale'. Game-server's GET /internal/seat-status?gameType=&tableId=&seatIndex= (service-key authed) lets the reconciler distinguish "still seated, just deaf heartbeat" from "really gone."

boot_reconcile.scan + boot_reconcile.reestablish + stale_reconciliation.scan + stale_reconciliation.action spans gate the telemetry.

D6. Skill wireMessageMap lifts game-specific WS shapes out of core/ ​

Every skill exports wireMessageMap: WireMessageMap<TAction, TWire> with builders sit(seatIndex) → TWire, stand() → TWire, action(TAction) → TWire, plus a per-skill extras bag. The orchestrator (apps/hosted-agent-service/src/agent-connection.ts) calls these instead of inlining game-specific JSON.

TWire parameterizes against the server's wire union (poker:cash-nlhe and poker:sng both bind to @numero/types's PokerClientMessage). A skill that produces a malformed wire shape fails compilation.

Skill loader at core/skill-loader.ts resolves a skillId string (e.g. 'poker:cash-nlhe', 'poker:sng', 'poker:_fixture') to the matching Skill via dynamic import — adding a new skill is one new branch, zero changes to core/. The T17 grep invariant (scripts/check-hosted-agent-core-boundary.sh) gates the discipline structurally: core/ cannot contain game-specific literals ('fold'/'check'/'call'/'bet'/'raise', sit_down, post_blind, preflop, pot odds).

The fixture skill at skills/poker/_fixture/ is the structural canary — instantiable + drivable without modifying any file in core/.

D7. Multi-agent-per-owner-per-table is admin-gated, default off ​

The structural capability is real — each agent has its own playerId=agentId row in table_players, independent chip balance, separate WS. The production policy is "one agent per owner per table." Toggle via the global runtime setting allowMultipleAgentsPerOwnerPerTable in game_settings (default false).

/seat queries this on every seat attempt; when false, it joins hosted_agent_seats → hosted_agents to count current seats by owner_user_id for this tableId and returns 409 agent_seat_owner_limit_reached if any exist.

D8. Audit ledger writes are durable ​

recordInferenceCharge is wrapped by recordInferenceChargeWithRetry (apps/hosted-agent-service/src/core/ledger-write.ts) with 3 bounded attempts (100/200/400ms exponential backoff). Migration 0043 adds UNIQUE INDEX agent_inference_charges_turn_uq on (agent_id, hand_number, action_index); the query uses .onConflictDoNothing() returning the existing row's id on conflict so retries are idempotent.

Migration 0044 flips the agent_id FK from ON DELETE CASCADE to ON DELETE SET NULL + drops notNull() — deleting a hosted_agents row preserves the audit history (only agent_id nulls out; owner_user_id and every other field stays). The audit trail outlives the agent record.

D9. Charge-alias gaps are recorded, not invented ​

chargeChipsForAlias() returns { chargeChips, missing }. When an alias has no MODEL_CHARGES entry, it returns { chargeChips: 0, missing: true } — never invents a default rate. The ledger row writes charge_chips=0, error_code='charge_alias_missing', success=true (the inference itself worked; only the billing axis records the config drift). agent.charge_alias_missing=true span attribute makes the drift queryable in Dash0.

Rejected alternatives ​

  • Leaving the broken path in place. The hosted-agent product surface is load-bearing for the platform. Three weeks of "deploy is broken" was already too long.
  • Extending table_players with nullable agentId instead of a separate hosted_agent_seats table. Considered for simplicity. Rejected because table_players is owned by BaseTable's persistence boundary contract; adding a runtime-only column would have leaked agent concerns into the cash code path.
  • Keeping the /deploy name. "Deploy" conflates "save the prompt" (CRUD) with "seat the agent" (operational). The rename to /seat matches the underlying semantics.
  • Per-table skill/model overrides. Considered. Rejected — the agent record is the unit of strategy choice; per-table overrides add a configuration matrix without a corresponding user need.
  • Configurable auto-stand-up threshold. Considered. Rejected — N=10 is a sane platform-wide default; configurability adds tuning surface without surfacing user value. Plan amendments can lift it if real data shows the threshold matters.
  • Agent-side wallet. Considered for the leave path (so auto-stand-up wouldn't need the owner's wallet to be linked). Rejected because it doubles the custody surface — every agent would need its own signing key — and the operational risk (lost-keys edge case during the interim window before custody-layer ships) is acceptable.
  • Deferred rename. Considered keeping /deploy until custody-layer ships. Rejected — the rename's cost is one PR, the cost of carrying a misleading name through every follow-up is higher.
  • Blocking on custody-layer. Considered making bulletproof-hosted-agents depend on the custody-layer plan landing first. Rejected — custody-layer is its own arc; this plan defines the seam (resolveForcedWithdrawDestination) that custody-layer will swap when it ships.

Files of interest ​

See also ​