skills·poker Docs

Appearance

Backend OTel Bootstrap Hotfix ​

Status: shipped + verified live on stg. Six of eight backends emitted zero telemetry in every deployed environment; all 8 now boot bootstrap-first, guarded by a CI check + a hermetic export harness. Falsification hardened the guard against three false-greens.

The bug ​

Production runs start, which booted dist/index.js — the file that never calls initTelemetry — while the correct bootstrap.ts (which inits OTel then dynamically imports the app) was only wired into dev. Every span became a NonRecordingSpan and was dropped at creation: the code ran, the instrumentation ran, nothing left the process. It failed quiet, so the whole money path (both banks, auth-server, game-server) was dark for weeks — and it defeated the plans built on top of it (the alert pack was wired to spans that never arrived).

Found by telemetry-driven smoke, not by any local check. A hand was played on staging and the agent read stg; game-server logged every deal and exported zero spans. No unit test, typecheck, lint, or green CI run could have caught it — the code and config were both correct. Only "did the span arrive in the deployed environment?" catches this class.

What was decided ​

Bootstrap-first everywhere, enforced two ways. Every backend's production entrypoint is a bootstrap.ts that calls initTelemetry (+ assertTracerRecording) and reaches the app only through a dynamic import("./index.js"). The build must emit dist/bootstrap.js (a second tsup entry) and still emit dist/index.js (the app-entrypoint.sh:61 pre-built sentinel; dropping it re-arms a known Fly build-at-boot outage). Enforcement:

  • check-otel-bootstrap (scripts/check-otel-bootstrap.mjs, in pnpm typecheck) — a fast static scan asserting init-before-app-import + both build entries. It derives its covered set from apps/*/fly.toml (the Fly-deploy signal), so a new deployed backend is checked automatically — not a hardcoded list that drifts.
  • test/otel-bootstrap/ — a hermetic prod-boot harness that builds + boots each backend against a local OTLP stub and asserts a span actually leaves the process. Runs in ci.yml (signal); the static scan is the deploy gate.

Key tradeoffs accepted ​

  • Chose bootstrap-first over initTelemetry-in-index.ts. The latter is subtly lossy: ESM hoists index.ts's static imports above the init call, so auto-instrumentation patches nothing (manual spans survive, HTTP/DB spans degrade — the exact state hosted-agent-service was in). Bootstrap-first is the only shape that preserves auto-instrumentation.
  • Kept the tsup/tsc build-system split (six services on tsup, tournament-orchestrator on tsc) rather than unifying — the CI check holds the invariant both must satisfy. That split is the root cause of why the drift stayed invisible, but unifying six build configs was a larger, riskier change than lighting the money path needed.
  • CI cannot prove Dash0-stg ingest specifically (no deployed environment in CI) — so deployed-catalog verification is a post-deploy manual smoke, backstopped by assertTracerRecording in all 8 bootstraps (a silent-init failure is loud at boot).

Falsification hardened the guard ​

The goal-flip found the plan's own deliverable weaker than attested, all fixed:

  • H1 — the scan rejected initTelemetry in index.ts but not a static import … from "./index" in the bootstrap (same ESM-hoisting failure, different syntax).
  • H2 — the export harness was excluded from CI, so the sole CI gate was the foolable static scan; wired it into ci.yml (and then caught that it needed a standalone vitest config, because the @numero/test package's global setup requires a Hardhat node the stackless CI job lacks).
  • H3 — the guard's service list was a hardcoded duplicate of deploy-fly.yml; service #9 added to deploy but not the list would ship dark. Now derived from apps/*/fly.toml.

References ​