skills·poker Docs

Appearance

Bulletproof Persistence ​

Status: Phase 2 shipped (engine serialization + @numero/redis foundation). Phases 3–6 outstanding. This page tracks the current-state architecture and gets extended at each phase close.

What it is ​

A two-tier persistence model for in-game state. Pre-Phase-2, the only durable layer was Postgres via persistAtBoundary() — fine at hand boundaries, fatal mid-hand. Two surfaces motivated the lift:

  1. Bug 15 — concurrent seat custody divergence. The PM-reserve hold lived in game-server memory; deposit_intents lived in Postgres; no shared store mediated the admin-server ↔ game-server handshake. 9-agent concurrent dogfood produced custody freezes 1-of-9 times.
  2. Mid-hand restart voids the hand. Engine state, OHH-builder progress, current actor, pending blinds — all in process memory between deal and showdown. Any restart (deploy, OOM, crash) killed every in-flight hand.

Both surfaces are the same architectural gap: hot ephemeral cross-process state living in one process's memory with no durable mirror.

The framing ​

In poker, a turn is a hand. State splits into two tiers:

  • Inter-turn (in-hand) — turnstore. A dedicated Redis container (separate from Thirdweb Engine's redis per ADR D9). Holds three lifecycle classes:
    • turn:state:<tableId> — engine + OHH-builder + actor + pending blinds. Deleted at hand-end commit.
    • tournament:state:<tableId> — orchestrator state that persists across turns. Deleted at tournament close.
    • reserve:<tableId>:<seatIndex> — sub-turn seat-claim handshake. Deleted on confirmCredit or TTL.
  • Post-turn (post-hand) — Postgres. Committed boundary state via the existing persistAtBoundary contract. Unchanged.

Eviction is explicit DEL on Postgres commit success, not TTL-only. TTL (7d default) is the backstop covering prolonged downtime.

Why a dedicated turnstore (not the shared Engine Redis) ​

Per ADR D9, shared maxmemory-policy instance-globally couples our fund-affecting state durability to Engine's operational needs. The current noeviction default is safe, but any future change for Engine would silently apply to us. Splitting removes the coupling at the cost of one extra docker-compose service.

What's already built (Phase 2) ​

  • New @numero/redis workspace exporting getTurnStore, claimReserve/confirmCredit/releaseReserve, writeTurnState/readTurnState/deleteTurnState, plus version-skew enforcement helpers.
  • Three Lua scripts for the PM-reserve handshake (atomic HSETNX, atomic check-then-DEL).
  • Engine toJson()/fromJson() plus 8 subobject serializations, round-trip-tested against 14 representative fixtures.
  • turnstore composable.env contract + components + dependency wiring on game-server and admin-server (env wiring superseded by the Doppler migration 2026-06-10 — turnstore is now declared in the committed root compose with TURN_STORE_URL from Doppler; see How env vars work).

Rehydrate — COMPLETED (2026-07-06) ​

The design's READ side shipped: the boot reconciler (a no-op stub since Phase 3b) is live, and a game-server restart now resumes live play — tournaments in place, cash hands mid-hand — instead of voiding/rewinding.

Key mechanics: the staged-snapshot registry bridges the reconciler and the table's async construction (consumption is AWAITED — loadOrCreate returns before the ready chain runs); the resumed hand gets fresh room bookkeeping (hand span, checkpoint counter, a partial-OHH remainder); outcome span persistence.rehydrate with resumed | floor_no_snapshot | floor_version_skew | floor_unconsumed | failed.

What's coming (remaining) ​

  • Phase 4 — admin-server reserve integration in /api/deposit, POST /admin/unfreeze/:tableId + freeze_audit table.
  • Phase 5 — integration green-up, perf benchmarks (p99 checkpoint <5ms, boot <2s, memory <50MB).
  • Phase 6 — docs + drained-deploy companion + decision-page final pass.

Performance ​

(To be filled in at Phase 5 close.)

Key invariants ​

  1. The turnstore is the source of truth for in-flight hands.
  2. Engine's redis instance is separate; never share keys.
  3. Version-skew at boot → refuse + alert. Operator picks one of three runbook paths (migrate / FLUSHDB / roll back game-server).
  4. Redis-down at boot → game-server refuses to listen.
  5. Custody invariant continues to fire on real divergence — restart-tolerance is a NEW branch, not a relaxation.