Bulletproof State

Date: 2026-04-17 Status: Implemented

The Invariant

For every table, at every reconciliation checkpoint:

pool[tableId][token] (on-chain) == Σ derived_ledger(player, tableId) (folded from events)

derived_ledger(player, tableId) is computed by replaying on-chain history:

derived = Σ Deposited events
        − Σ WithdrawalExecuted events
        + Σ hand_players.net_chips (for completed hands since last withdrawal)

The chain is the source of truth. Ponder is a Postgres projection of the event log — a speed optimization, not a dependency. If Ponder goes dark, we fall back to direct RPC reads and keep operating (degraded but correct).

Why This Plan Exists

Chain-of-Custody 2 shipped a functional sit=deposit / stand=withdraw UX but regressed the invariant during multi-player testing. After several game-server restarts the DB showed 400 CTN on-chain and 0 CTN in room_players.chip_balance for the affected wallets. Silent drift. No alert. The game-server kept serving on a wrong view of reality.

Root causes:

room_players.chip_balance was written as a primary record by multiple code paths (credit-chips, sit-down, reconstructFromDb)
Ponder didn't index the unified Deposited / WithdrawalExecuted / HandSettled events (only legacy per-game events)
No startup reconciliation — the server trusted whatever was in the DB
No invariant assertion anywhere

Bulletproof-state is the architectural response. Short version: events are truth, intent tables are the crash-recovery anchor, the reconciler is the only authority.

The Four Load-Bearing Decisions

1. Intent-outcome pattern on every on-chain flow

Two new tables:

deposit_intents
  idempotency_key PRIMARY KEY  = hash(permit)
  intent_kind                  = 'sit_down' | 'top_up' | 'agent_sit'
  wallet, table_id, amount, room_code, seat_index, user_id
  status                       = pending | submitted | confirmed | failed
  tx_hash, event_tx_hash, event_log_index, failure_reason

withdrawal_intents
  idempotency_key PRIMARY KEY  = hash(player, tableId, historyHash)
  player_id, table_id, wallet, amount, history_hash, room_code
  status                       = pending | deferred | submitted | confirmed | failed
  settle_tx_hash, withdraw_tx_hash, event_tx_hash, event_log_index

The intent row is the recovery anchor. It's written before the tx is submitted; a crash at any point leaves enough state for the reconciler to finish or cancel it. Idempotency keys are content-derived so replay detection needs no prior DB round-trip — same permit → same key → cached result.

deferred is withdrawal-only. A player standing up mid-hand gets a deferred intent; the hand-end hook promotes it to pending and submits. The failure sweep explicitly excludes deferred rows so long hands don't false-fail.

2. Ponder is the fast path. Direct RPC is the fallback. Both operational at all times.

After a deposit tx submits, admin-server:

Polls ponder.deposits for up to 5s (typical indexing: 1-3s).
If Ponder has it, mode = 'ponder', mark confirmed.
If Ponder is empty, slow, or unreachable, call publicClient.getTransactionReceipt(txHash) and parse the Deposited log from the receipt. Same event_tx_hash + log_index extracted, mode = 'rpc-fallback', mark confirmed.

The response to the caller includes mode so observers can see when we're degraded.

Freeze triggers are NOT the same as Ponder outages. Ponder-down is a WARN + reconciler.degraded=1 metric, not a freeze. Only RPC-down (chain itself unreachable for >30s) freezes. This distinction is load-bearing — otherwise every Ponder restart kicks every player out. See chaos-7a for the proof.

3. Freeze semantics: finish-hand, refuse-new, allow-standups

When the reconciler detects divergence (pool != derived):

rooms.status        = 'frozen'
rooms.freeze_reason = 'divergence' | 'rpc_down'

Behavior matrix while frozen:

Operation	Allowed?
Current hand	finishes normally
New hand	refused
New sit-down	refused (`409 table_frozen`)
New top-up	refused (`409 table_frozen`)
Stand-up	allowed (escape hatch)
Admin reconcile	runs invariant, updates status
Admin unfreeze	re-runs invariant; clears flag if now `ok`

4. Writes to `room_players.chip_balance` go through the reconciler

The intent-outcome pipeline terminates in the reconciler. When a Deposited or WithdrawalExecuted event matches an intent, the reconciler (a) marks the intent confirmed and (b) recomputes the derived ledger for that (player, tableId). The cached room_players.chip_balance is overwritten from the recomputed value.

Legacy write paths removed:

POST /internal/credit-chips (game-server) + POST /settlement/confirm-deposit (admin-server) + the poker /api/deposit/confirm proxy — all deleted in Phase 7.
_handleWithdraw (game-server shared/settlement) — deleted, replaced by standUpPlayer → /settlement/withdraw with idempotencyKey.

The design intent is that room_players.chip_balance has exactly one authoritative writer. In-hand chip movement is in-memory in the ParticipantManager (pool doesn't change); the DB cache is snapshotted at hand-end via the existing doCheckpointParticipants hook — safe because the reconciler will overwrite with the derived value on the next check.

Deposit Flow (Intent-Outcome)

Withdrawal is the mirror: standUpPlayer computes hashWithdrawalKey(player, tableId, historyHash), admin-server submits settle + withdrawFor, reconciler matches the WithdrawalExecuted event.

Chaos Test Matrix

The invariant is validated by a runnable test harness — 10 scenarios, each exercises real infrastructure (Base Sepolia + Engine + Ponder + Postgres). Not a mocked unit test — the actual stack, with real on-chain transactions.

#	Scenario	What it proves
baseline	Pool read + invariant primitive wired	The `/settlement/pool/:tableId` endpoint works and `checkInvariant` runs
1	Happy path deposit	Intent transitions pending → submitted → confirmed; ponder.deposits row appears; invariant holds
2	State machine correctness	Every transition populates the right fields: tx_hash, event_tx_hash, event_log_index, updated_at monotonic
3	Double-submit replay	Same permit twice = same txHash, exactly 1 intent row, exactly 1 on-chain tx
5	Bad permit failure path	Expired permit → clean failure, no orphan state, balance unchanged, no ponder row
6	Stale `room_players` detection	`checkInvariant` reports delta + per-player breakdown on divergence
7a	Ponder down, RPC up	Deposit confirms via `getTransactionReceipt` fallback; table NOT frozen
8	Manual divergence → freeze → unfreeze	Full freeze lifecycle; frozen tables refuse deposits; admin unfreeze clears when divergence resolved
11	Withdrawal happy path	Deposit → withdraw round trip returns pool to zero; invariant holds

Scenarios 4, 7b, 9, 10 are written as skeletons but not implemented — they need fault-injection infrastructure (CHAOS env flags), RPC-down monitor, or full game-server WS flow with active hands. Tracked as follow-ups.

Freshly Redeployed Stack

This plan was a fresh start:

Diamond redeployed: 0xc03e7611f7df7a792c04662e75dc05d938d9baff
ChitinToken redeployed: 0xf0fc1381bb21e98aa4d9ab6de4910154f1c3cf3d
Ponder DB wiped, handlers rewritten for the unified events only
room_players, rooms, poker.hands, hand_players, positions, position_events, table_ledger, rake_reconciliations all truncated
Authentication state (users, user_auth, wallet_addresses) preserved so existing test accounts still work
Previous deploy (with the stranded SHQQ3A 400 CTN) is abandoned on-chain — not migrated

Key Files

Path	What
`packages/db/src/schema.ts`	`depositIntents`, `withdrawalIntents`, `rooms.freezeReason`
`packages/db/src/intent-keys.ts`	`hashPermit`, `hashWithdrawalKey`
`packages/db/src/intent-queries.ts`	CRUD helpers + caller policy
`apps/admin-server/src/routes/settlement.ts`	`/settlement/deposit`, `/settlement/withdraw` with intent-outcome
`apps/admin-server/src/routes/admin.ts`	`/admin/reconcile/:roomCode`, `/admin/unfreeze/:roomCode`, inline fold
`apps/game-server/src/shared/reconciler/fold.ts`	Ponder SQL path + RPC `getLogs` fallback
`apps/game-server/src/shared/reconciler/invariant.ts`	`checkInvariant(tableId, deps)`
`apps/game-server/src/shared/reconciler/onchain.ts`	`makeAdminPoolReader({ adminUrl, serviceKey })`
`apps/game-server/src/shared/room-handler.ts`	`handleSitDown`, `handleTopUp`, `standUpPlayer` all send `idempotencyKey`
`apps/ponder/src/index.ts`	Unified event handlers only (legacy deleted)
`scripts/test-harness/chaos/`	10 chaos scenario files + baseline + shared helpers

References

Sit=Deposit, Stand=Withdraw — the UX layer this invariant sits under
Chain of Custody — zero-decimal CTN, pooled tables, unified settlement — the foundation
Greg Young, "Event Sourcing" (2010)
Martin Fowler, Event Sourcing
Stripe engineering — "Idempotency keys"

Bulletproof State ​

The Invariant ​

Why This Plan Exists ​

The Four Load-Bearing Decisions ​

1. Intent-outcome pattern on every on-chain flow ​

2. Ponder is the fast path. Direct RPC is the fallback. Both operational at all times. ​

3. Freeze semantics: finish-hand, refuse-new, allow-standups ​

4. Writes to room_players.chip_balance go through the reconciler ​

Deposit Flow (Intent-Outcome) ​

Chaos Test Matrix ​

Freshly Redeployed Stack ​

Key Files ​

References ​