skills·poker Docs

Appearance

Bulletproof Observability ​

Date: 2026-05-12 (closed) Status: Implemented Supersedes: the original telemetry, frontend-OTel, and continuous-playtest observatory planning streams, all folded into one unified plan.

The Decision ​

Apply the bulletproof discipline (runtime contracts in code, fail-loud invariants, structural prevention via test trajectory) to the entire observability surface of apps/poker/'s production stack — front-end click → admin-server → game-server → ponder → DB — and ship the dev-time feedback loop (hand envelopes, anomaly detection, NPC-driven continuous play, admin-UI dashboard) that uses that substrate to catch and surface bugs the system wasn't told about.

The bug class this plan structurally prevents: "a regression in telemetry / wire-format / engine output is silently shipped because no test asserts the invariant." The seed example is the OHH all-in-runout regression: the poker engine deals 5 community cards on all-in pre-flop, but autoAdvance overwrites lastBroadcast each iteration so only the final showdown survives — OHH records 2 rounds, DB shows cc_col=5. No test caught it; humans only noticed when they happened to look.

Eight committed decisions ​

  1. Per-service telemetry contract + tripwire (6 services). Every service that emits manual OTel spans declares its canonical span list in apps/<service>/src/telemetry-contract.ts. scripts/check-telemetry-contract.ts greps every startActiveSpan('X', ...) / startSpan('X', ...) literal in each service's source tree and fails the build if (a) a contract entry has no matching callsite OR (b) a callsite has no contract entry. Wired into pnpm typecheck. 6 services enforced: game-server, admin-server, hosted-agent-service, ponder, frontend poker, playtest-npc-service. Adding a new service: write the contract file + add to the CONTRACTS array in the same PR.

  2. Mock-exporter test endpoint pattern. Game-server bootstrap installs a triple-gated in-memory ring-buffered span exporter (10k capacity) when NODE_ENV !== 'production' + ENABLE_TEST_AUTH === 'true'. POST /internal/test/spans-since/:ts (service-key auth) returns finished spans since the requested epoch-ms timestamp. The vitest harness uses this to assert "every contract-declared span fired during one hand" without round-tripping through Jaeger. Bootstrap also runs assertTracerRecording() after initTelemetry() — opens a synthetic startup.assertion span, logs error if isRecording() returns false (catches silent OTel-init failures where misconfigured exporter env produces NonRecordingSpans).

  3. hand_envelopes as a new table (not a JSONB column on poker_hands). One row per completed poker hand with primary key (table_id, hand_number). Columns: ohh (canonical document), spans (test/dev only — empty in prod), chip_reconciliation (per-participant + sum check), anomalies (rule output array), trace_status (complete / partial / dropped_on_disconnect), assembler_error (non-null on write failure). Cross-table content is why this is its own table — couples to neither OHH evolution nor the engine's hand_players row shape.

  4. Anomaly rules as TS functions. Each rule in apps/game-server/src/observatory/rules/*.ts is a pure function (envelope) => AnomalyResult | null. Registered in index.ts:ALL_RULES. The assembler iterates with per-rule try/catch (F14) — a throwing rule is captured as an anomaly with error: 'rule_eval_failed', other rules continue. v1 ships 4 rules: ohh_runout_round_count_mismatch, chip_reconciliation_delta, span_topology_action_count, action_amount_chipflow_mismatch. Adding a rule: drop a .ts file, export an AnomalyRule, register in index.ts.

  5. Admin-UI dashboard at /observatory reading envelopes directly (via the admin app's /api/observatory/* proxy to admin-server's /observatory/* routes, per the "no DB from Next.js" convention). List view paginated by created_at DESC with tableId filter + flagged-only toggle. Detail view: anomalies panel + chip reconciliation table + OHH JSON + recursive span tree, side by side. Dev tool, not prod monitor.

  6. NPC service evolution — apps/playtest-npc-service/ emits npc.session (root per instance) + lifecycle spans (npc.connect, npc.deposit, npc.stand_up) + per-turn npc.decision.<actionType> spans. The decision span captures heuristic + inputs (legal_actions, chipRange, chosen action). NPC injects W3C traceparent into the action WS message; server-side handleAction extracts it and adds a SPAN LINK on poker.action — preserving the server-internal parent (poker.hand) while making the cross-process NPC→engine relationship queryable. On-demand via pnpm playtest:start --table=X --count=N — not a separately-deployed service in v1.

  7. useGameDispatcher as canonical UI instrumentation surface. Shipped under bulletproof-pregame-ui; extended in Phase 2 with an opt-in DispatchOptions.spanName override per dispatch call. In-game poker actions (fold/check/call/bet/raise) dispatch {type: 'action', action: subtype} with spanName: ui.action.<subtype> so the WS payload stays server-compatible while the span name reflects the subtype. UI_POKER_ACTION_TYPES enumerates the 12 concrete action types for the parameterized test. HTTP-dispatched flows (ui.deposit.submit, ui.topup.submit, ui.withdraw.submit, ui.faucet.request) emit their own per-component spans. Frontend contract grep scoped to apps/poker/src/ + apps/poker/lib/ only — App Router pages excluded by convention (spans live in hooks/components, not pages).

  8. "Telemetry never blocks the user" as a structurally-enforced invariant. Try/catch wraps every span boundary at four points:

    • Frontend dispatcher (useGameDispatcher swallows startSpan exceptions)
    • Backend manual span wrap (flow.ts poker.* spans use try/finally; setImmediate catches in the envelope dispatch)
    • Envelope assembly (setImmediate ordering — assembler runs strictly after hand_result broadcast)
    • Anomaly rule evaluation (per-rule try/catch with rule_eval_failed capture)

    Test discipline locks this: F11 (frontend), F12 (backend), F13 (envelope latency), F14 (rule throw) all have authored tests. Telemetry failures result in trace_status='partial' envelopes, never user-visible errors.

What shipped — substrate layers ​

Trace topology — one connected story ​

A poker hand played end-to-end now produces a queryable cross-process trace:

The dashed line is the cross-process span link: NPC-driven actions share a trace with their decision context AND the server-internal hand trace. The dashed setImmediate arrow is the F13 invariant — envelope assembly runs strictly after broadcast, never before, never blocking.

Trajectory close state ​

26 trajectory rows across the four phases:

  • Passing (structural / unit): T3, T4, T5, T6, T7, T13, T14, T15, T16, T19, T23, T26 — 12 rows. Tripwire enforces every PR; rule fixtures pass on every test run.
  • Skipped (docker-stack-dependent): T1, T2, T8, T9, T10, T12, T18, T20, T21, T24, T25 — 11 rows. All authored as .skip()'d harness scenarios; flip to passing on first manual stack run with ENABLE_TEST_AUTH=true + relevant feature-flag dependencies.
  • Deferred: T11 (admin-app vitest infra), T17 (F5 banner), T22 (PII allowlist) — 3 rows. Each captured as follow-up plan trigger with explicit unblock dependency.

The HARD close criterion is T10: a deliberately re-seeded engine regression is flagged by the rule-based detector within ≤3 NPC-driven hands. Currently .skip()'d but mechanically end-to-end wired — the rule is in place, the envelope assembler runs, the NPC service plays, and the OHH-runout bug is intentionally unfixed (per ADR) so any all-in-preflop hand reproduces it. Flips to passing on first docker run.

The bug we left in on purpose ​

autoAdvance in apps/game-server/src/games/poker/engine.ts overwrites lastBroadcast each iteration. On all-in-preflop hands that cascade flop+turn+river+showdown in one autoAdvance call, only the final showdown broadcast survives. OHH records Preflop + Showdown (2 rounds) even though DB's cc_col=5 confirms 5 community cards were dealt.

This bug is intentionally unfixed in this plan — fixing is its own future plan. The TELEMETRY_OHH_RUNOUT_REGRESSION_REENABLE env flag is forward-compatible scaffolding: when a future plan ships the fix (collect broadcasts in an array, return them all), the flag becomes the explicit re-enable path for regression testing. Until then, the flag is a no-op (because the bug is the default) and the observatory's ohh_runout_round_count_mismatch rule fires on any all-in-preflop hand.

This is what closes the plan honestly: the system catches the seed bug not because we taught it about that specific bug, but because the chip-conservation + span-topology + OHH-vs-engine cross-checks pick up the symptom from multiple angles. Future engine regressions of similar class fall into the same rule shape and become observable without code changes.

Out of scope (each a future plan inheriting this substrate) ​

  • Arena instrumentation — arena.round / arena.action backend spans, arena UI spans. The system this plan builds makes it a one-PR exercise (declare contract → register → write the spans).
  • Browser OTel rebuild for admin / dealer / frontpage / poker-next — same vendor-OTel rip-out pattern as poker got. Queued behind @numero/telemetry-web extraction at 2nd consumer.
  • @numero/telemetry-web workspace package — promote apps/poker/lib/telemetry.ts when a 2nd UI consumer materializes.
  • AI-driven RCA — Claude reading envelope diffs and proposing root causes. Needs this plan's substrate; own future plan.
  • Logging contracts — same shape applies to getLogger(...) calls. Bulletproof-logs is a follow-on if needed.
  • Production paging / SLOs — observatory is a dev tool, not a prod monitor.
  • OHH bug fix — left as the close-criterion seed. Own future plan.
  • PII allowlist automation — F10 deferred to manual review; revisit when reviewer load grows.
  • Poker UI Foundation Rewrite — useGameDispatcher lives in the framework layer of that tree-shape; bulletproof-observability extends it with the spanName override.
  • Bulletproof Pre-game UI — useGameDispatcher as canonical instrumentation surface was authored there; this plan extends + parameterizes coverage.
  • Bulletproof State Machine (UI) — deposit_progress / withdraw_progress wire messages drive the state slices the observatory's anomaly rules cross-check.
  • Poker OHH Integration — OHH document is the per-hand fairness artifact; envelope binds OHH + spans + chip reconciliation into one queryable unit.

Status ​

Implemented and committed across 4 phases (2026-05-12). Substrate is live; the test scenarios await a first full-stack run for end-to-end verification.