Tournament Durable Orchestration

Status: LIVE ON STAGING (2026-07-14), proven under kill-testing (2026-07-15). Production cutover + retiring the in-process orchestrator are on hold behind tournament-deal-authorization, which reverses the direction discipline below — do not build on E-TDO-3 as stated. · Full ADR: .indusk/planning/tournament-durable-orchestration/adr.md (see Amendment 1)

What was decided

The tournament lifecycle + money orchestration moves out of the game-server's real-time process into a durable workflow — implemented with the DBOS library, durable state in our existing Cloud SQL (a dedicated tournament_orchestrator database, never the app DB), running as a dedicated apps/tournament-orchestrator service. The game-server keeps only the real-time Engine + hand loop.

Why: the tournament lifecycle is a long-running, restart-surviving, exactly-once workflow that was implemented inside a real-time process with hand-rolled durability (turnstore snapshots, boot rehydrate, payout_intents + reconciler sweeps). Durable-execution engines solve exactly that class. A workflow's state is journaled per step, so a restart resumes from the last completed step — there is no resumeFromBoot, no floor-stack archaeology, no void-vs-setTimeout timing bet.

The real-time / durable split

Direction discipline (E-TDO-3) — REVERSED 2026-07-16, see the amendment below. As originally decided: the game-server never asks the workflow permission per hand; it drives the real-time loop autonomously and reports each committed hand boundary up as a signal, and the workflow reacts with the next coarse command. That is what SHIPPED and what runs on staging today; it is not what we are building toward.

Amendment: the table now asks permission to deal

A table cannot answer "should I deal the next hand?" on its own — in an MTT the answer is often "no, your table is breaking." Breaks, balancing, hand-for-hand at the bubble and synchronized levels are all inexpressible under autonomy. E-TDO-3 optimized for a single-table SnG and mistook it for the general case.

Autonomy also caused essentially every defect the kill-testing found: a dropped boundary push left the workflow's view stale forever; it could not tell "the game stopped" from "I can't hear the game" and force-concluded a healthy tournament on stale stacks; a settled tournament kept dealing over its own closed books; and with the orchestrator down the table dealt 23 hands at level-1 blinds — a materially different tournament than advertised — then jumped to the caught-up level.

So: a durable-owned table must be AUTHORIZED to deal each hand (blinds ride the authorization, so they cannot go stale), and a tournament level becomes an amount of time PLAYED — the clock stops when the tournament stops and resumes where it left off. An unreachable authority means the tournament PAUSES; it does not invent its own game. The principle: the goal is not "keep running when a dependency is down" (that keeps running wrong) but "stop cleanly and resume exactly where you were."

Unchanged: game-server still writes the hand (OHH, hashes, deltas, chip checkpoint) in one local transaction BEFORE reporting — it is the only process that can build them, it is a BaseTable.persistAtBoundary framework contract shared with cash + arena, and committing locally first keeps the hand durable when the orchestrator is away. Cash stays autonomous: no clock, no levels, no eliminations, nothing to orchestrate.

See .indusk/planning/tournament-durable-orchestration/adr.md Amendment 1 and the tournament-deal-authorization plan.

The contract (framework-agnostic, MTT-aware)

Every command and signal is tournament-scoped and table-addressed (E-TDO-6) — a single-table SnG is the N=1 case, so multi-table-tournaments becomes a parent-workflow follow-on, never a contract rework.

Direction	Call	Idempotency key
workflow → game-server	POST /internal/tournament/seat-and-deal	{t}:seat:{table}
workflow → game-server	POST /internal/tournament/set-blinds	{t}:blinds:{table}:{level}
workflow → game-server	POST /internal/tournament/teardown	{t}:teardown:{table}
workflow → cage	POST /api/withdraw (payoutIntent)	{t}:pay:{position} + on-chain payoutId/payout_marker
workflow → cage	POST /escape/:tableId	{t}:escape:{table}
game-server → workflow	POST /internal/tournament/hand-boundary	bnd:{t}:{table}:{hand}
cage → workflow	POST /internal/tournament/registration-paid	reg:{t}:{userId}
cage → workflow	POST /internal/tournament/payout-confirmed	payconf:{t}:{position}

Signals ingress over service-key HTTP into DBOS.send with an idempotency key, so a retried POST delivers exactly once (proven durable across a kill -9 in the Phase 0 spike).

Portability discipline (the cheap exit)

DBOS/Temporal/Restate share a programming model; they differ in API and operational weight. The orchestrator is written as plain functions with DBOS at the edges so a later re-platform is drain-and-cut plus a thin annotation swap:

• @DBOS.transaction is forbidden (the one DBOS primitive with no Temporal equivalent).
• No wall-clock setTimeout — durable timers (DBOS.sleep / recv-timeout) only.
• Enforced by apps/tournament-orchestrator/scripts/check-portability.mjs (E-TDO-1, proven to bite on planted violations).
• src/commands.ts has no DBOS import — it is ordinary idempotent HTTP, the same shape a Temporal activity would call.

Money steps (E-TDO-4)

The durable-step retry replaces the bespoke payout_intents reconciler sweep, but the on-chain single-use guards (payout_marker PDA / EVM payoutId, custody claim_marker) remain the idempotency floor — belt and braces, never one alone. assertConservation (Σ payouts == pool) runs before any money moves; the workflow awaits per-position payout-confirmed signals before terminal close.

Cutover (E-TDO-5)

Drain-and-cut per environment. Ownership is resolved once, at a tournament's first load, then stamped into tables.settings.orchestrationEngine — so a flag flip never moves an in-flight tournament (new tournaments route to the workflow; in-flight ones finish on the in-process orchestrator). A tournament never straddles both orchestrators. The same mechanism is the framework-migration path later (tournaments are short-lived — hours — so a swap never migrates live state).

The resolution has two layers (matching the PostHog-integration split — PostHog is the rollout/targeting layer, game_settings is the server-authoritative kill-switch):

1. game_settings.tournament_orchestration_engine — an explicit durable / in_process value ALWAYS wins. This is the emergency kill-switch, and how a whole environment is forced on/off (e.g. an isolated test env).
2. Otherwise the PostHog tournament_durable_orchestration flag (server-side local evaluation, keyed on tableId) decides — so a percentage rollout deterministically shifts whole tournaments onto the durable engine.

Fail-closed to in_process: PostHog unset/unreachable, or no game-setting override, → current behavior. A tournament can never be routed onto the durable engine by accident.

Ops

• DBOS Conductor (hosted console, outbound API key) is the ops cockpit: view live workflows, see where a step broke, one-click resume-retry. Because Conductor sees step metadata, step arguments carry only opaque ids (tournamentId/tableId/position) — never wallets or secrets (D7).
• Workflow state is also plain SQL (dbos.workflow_status, operation_outputs, …) + a programmatic admin API (listWorkflows/cancelWorkflow/resumeWorkflow), so the pure-infra fallback (SQL + Dash0 + internal admin page) needs no SaaS.
• Domain spans: tournament.workflow (provision) + tournament.command.* + tournament.signal.*, contract-registered. DBOS's native workflow/step spans ride the same OTLP endpoint.

Gotchas

• Cloud SQL + DBOS v4 SSL: the driver defaults sslmode=require to full CA verification, which fails on Cloud SQL's cert chain. Append &uselibpqcompat=true (libpq semantics: encrypt, don't verify) to DBOS_SYSTEM_DATABASE_URL.
• The domain root span marks provisioning, not the whole run — an hours-long in-memory span would never export on a crash. The durable execution itself is traced by DBOS's own spans.
• Deploy is inert until activated: nothing routes tournaments to the service until the Phase 4 flag flip. Activation checklist: apps/tournament-orchestrator/DEPLOY-WIRING.md.

What was rejected

• In-process + more hand-rolled durability — we keep rebuilding the same durable-step/reconciler/idempotency-key shape per money flow, and sharding breaks ownership.
• Restate now — adds a stateful broker tier to keep alive right before a fund launch.
• Temporal now — heaviest operational footprint before the approach is validated.
• DBOS inside the game-server — re-couples orchestration to the process we're making disposable.