Tournament durable orchestration — live on staging, and it falsified its own core assumption

2026-07-16

The tournament lifecycle now runs as a durable DBOS workflow in apps/tournament-orchestrator, live on staging since 2026-07-14. A tournament's identity — roster, level, who is out, who has been paid — survives game-server, and a dead tournament is a visible, resumable row instead of silence.

What shipped

• The durable workflow (registration → seat-and-deal → level clock → settle → pay → teardown), with the money steps' on-chain single-use guards (payout_marker / EVM payoutId) as the exactly-once floor.
• Two-layer cutover: a game_settings kill-switch over a PostHog rollout flag, fail-closed to in_process, with ownership stamped per tournament (a flag flip never moves an in-flight one).
• Registration-timeout refund: a tournament that never fills now escapes its pool to custody instead of stranding every payer's entry fee on-chain.
• Every step parks-and-retries rather than throwing — a dropped dependency (i.e. any deploy) used to kill the workflow permanently with players' money on the table.

What the kill-testing found

Killing the orchestrator mid-tournament proved the durable claim: DBOS recovered the workflow and the tournament completed and paid. Killing its dependency proved the rest of the design wrong:

• Boundary pushes are fire-and-forget, so an outage left the workflow's view stale forever (hand 1 → restart → hand 24).
• The workflow could not tell "the game stopped" from "I can't hear the game", and force-concluded a healthy tournament four seconds after recovery, paying on stale stacks.
• A settled tournament kept dealing for three minutes after close and check-pointed over its own closed books.
• With the orchestrator down the table dealt 23 hands at level-1 blinds, then jumped to the caught-up level.

Each was fixed by the same idea — decide from the committed DB, not from a droppable signal — and each was a patch over the same hole.

The decision that came out of it

E-TDO-3 is reversed. A table cannot answer "should I deal the next hand?" alone: in an MTT the answer is often "no, your table is breaking." A durable-owned table will be authorized per hand (blinds riding the authorization), and a level becomes an amount of time played — the clock stops when the tournament stops. An unreachable authority means the tournament pauses rather than inventing its own game.

Production cutover and retiring the in-process orchestrator are on hold behind that work: never cut production onto a contract we have decided is wrong.

Current state: Tournament Durable Orchestration.