testing

PURPOSE — Dev-only QA infrastructure that exercises the live engine under controlled conditions: AABB artifact proof tests with seeded determinism and tier-sweep matrix, a boss gauntlet that scripts deterministic boss-anchor damage to measure perf and balance per-boss, a headless stress-test harness that synthesizes worst-case loads and measures pure update cost without rendering, URL-activated dev scenarios that pre-configure a real run for visual / balance / artifact verification, and a hidden perf-test mode that boots cranked-load runs in an auto-restart loop and ships periodic snapshots to Supabase for overnight soak.

OWNS

• The scenario / matrix / weapon-gauntlet runner singleton (scenarioRunner), its seeded-PRNG install-and-restore protocol, manual-pump frame loop, baseline cache keyed by scenario type, verdict taxonomy (PROVEN | WEAK | NO_EFFECT | NOISY | BROKEN | ERROR) and scaling-shape taxonomy (flat | linear | exponential | inverted).
• The static A/B test list and ship / world / flow presets keyed per-artifact, plus the three matrix ScenarioTemplates (SWARM / SINGLE_TARGET / SURVIVAL).
• The boss-gauntlet harness, its mission-reuse + sandbox-reset isolation pattern, scripted boss-anchor damage primitive, weapon-stash + player-damage-disabled bundle, stuck-invuln watchdog bypass protocol, and per-mode pass / fail gates (smoke / balance / survival / stress).
• The boss-gauntlet type contract and the per-mode config table (BOSS_GAUNTLET_MODE_CONFIGS), boss run order (BOSS_GAUNTLET_ORDER), telemetry sample stride, and projectile-count perf gate.
• The scenario / matrix / runner type module — all data shapes consumed by the workbench UI, runner, and matrix sweep, with no runtime logic.
• The pre-built named test sequences (ARTIFACT_SUITE_T0, ARTIFACT_SUITE_T3, WEAPON_DPS_SWEEP, STRESS_SUITE) and their registry.
• The dev-scenario registry indexed by ?dev=<name> URL params and the shared baseRun builder that overrides fog / timer / ship durability before per-scenario tuning.
• The headless stress-test harness, its ten scenario functions, the synthetic enemy / event / weapon factories that drive them, and the per-scenario StressResult with avg / min / max / p95 / p99 frame-time stats.
• The perf-test-mode session state (_sessionId, snapshot index, run count, device context, active flags), the 30-second telemetry cadence, the cranked-load RunDefinition factory, and the perftest_start / perftest_snapshot / perftest_end event triplet.

READS FROM

• engine/core/state — live game, ship, world, camera singletons for direct mutation in stress tests and read-only telemetry sampling in the boss gauntlet.
• engine/core/config — PERF_FLAGS (perftest activation, isMobile, dprOverride), BUILD_VERSION, PERF_VERSION_TAG, CFG.MAX_XP_ORBS, CFG.MAX_PARTICLES.
• engine/core/render-diag — per-pass render-perf snapshot and spike-drain queue consumed by perf-test snapshots.
• engine/core/signals — Sig.on('boss_kill') subscription in the boss gauntlet per run.
• engine/combat/collision-resolver, engine/weapons/weapons, engine/weapons/bullets, engine/vfx/particles, engine/world/events, engine/world/xp-orbs — production update paths invoked directly by the stress-test harness without going through the real game loop.
• engine/vfx/boss-layers — getBossVfxPassMs() and bossVfxLayerCount() sampled into boss-gauntlet telemetry frames.
• data/artifacts, data/weapons, data/enemies, data/bosses, data/run-config — entity defs, scaling curves, the DEFAULT_RUN baseline cloned by dev scenarios and perf-test mode.
• window.__mission (sandbox mission handle), window.__dev (dev hooks), window.__effectTriggerCounts (effect-engine trigger map) — bridge surfaces consumed by the scenario runner.
• window.location.search — dev-scenario activation and override params (?dev=, ?weapon=, ?level=, ?ship=, ?kills=, ?tier=).

PUSHES TO

• window.__mission API: sandboxResetForTest, sandboxSpawnBoss, sandboxGrantArtifact, sandboxSetSpawnRate, setSpawnerEnabled, setWeapons, setWorldKnobs, setGodMode, patchShipStats, fullHeal, spawnEnemyAt, testSetManualPump, testPumpFrame — every sim mutation goes through the bridge, never direct world writes (except in stress-tests, which deliberately mutate singletons).
• The live engine state singletons in stress tests — direct writes to ship.hp, ship.weapons, world.enemies, world.events, world.particles, world.dmgNumbers, world.playerBullets, world.enemyBullets, game.phase, game.time, game._dt, game.timeDilation.
• The boss-gauntlet harness writes directly to the boss anchor’s hp field (and sets _dying on death) — bypasses the bullet / weapon / damage pipeline so kill rate is deterministic.
• metagame/services/analytics trackEvent — perf-test mode fires perftest_start, perftest_snapshot (every 30s), and perftest_end carrying device context, active flags, perf snapshots, top-5 worst spikes, and JS heap stats.
• console.log — tagged structured lines ([AABB_TEST], [AABB_TEST]_RESULT, [AABB_TEST]_SUMMARY) from the scenario runner, formatted stress-test result tables, and console.warn for unknown dev-scenario names.
• Listener callbacks registered via the runners’ onUpdate / onMatrixUpdate / onWeaponGauntletUpdate / setProgressListener — the dev workbench UI subscribes and renders progress / results.
• window debug surfaces: window.__stressReport (last stress report), window.__bossGauntlet (browser-console boss-gauntlet API), window.__sampler (live-tuning probe from telemetry).
• Return values: ABResult[], ArtifactMatrixResult[], WeaponGauntletResult[], BossGauntletRunReport, StressReport, RunDefinition from buildDevScenario / buildPerfTestRunDef.

DOES NOT

• Run in production. Every entry point is dev-only — the scenario runner expects the dev bridge surfaces, the boss gauntlet expects an injected sandbox mission ref, stress tests are imported by Vitest or the dev console, dev scenarios require the ?dev= URL param, and perf-test mode is gated on PERF_FLAGS.perftest.
• Render. The stress-test harness has no canvas, no WebGL, no draw calls — frame cost is pure update / sim. The boss gauntlet uses the engine’s existing rAF render loop (it doesn’t pump frames manually). The scenario runner’s matrix and weapon-gauntlet paths pump frames manually but yield to the browser per frame for live canvas repaint.
• Use the player’s weapons against the boss. The boss gauntlet stashes ship.weapons to a local, clears the array, and sets ship._gauntletPlayerDamageDisabled = true so artifact-tagged damage (Personal Space, Echo Generator) is also suppressed — scripted DPS is applied directly to the boss anchor’s hp.
• Use Math.random() during scenario runs without first installing the seeded Mulberry32 PRNG. Matrix and weapon-gauntlet paths additionally use try/finally to guarantee restoration, so a thrown exception inside a run can’t leak the seeded RNG into the rest of the page.
• Persist results between sessions. Every report (ABResult[], BossGauntletRunReport, StressReport) is in-memory only; perf-test mode is the one exception — it fire-and-forgets to Supabase via trackEvent with no local cache or retry.
• Render UI. Consumers subscribe via the runners’ update callbacks and render their own progress chrome; this module exposes no React components.
• Validate that referenced ids exist — test sequences reference scenario ids and artifact ids by string; dev scenarios accept weapon / ship strings without checking the data tables; the boss gauntlet trusts that BOSS_GAUNTLET_ORDER ids resolve in data/bosses.
• Define balance numbers. Tuning constants (NOISY_THRESHOLD = 0.20, TEST_SEED = 20260416, INIT_ENEMY_COUNT = 12, ENEMIES_PER_WAVE = 6, mode timeoutSec / playerDpsToBoss / enemyDamageMult) are testing-harness numbers, not game balance — they live in this module’s type / config files, not in the shared data tables.
• Assert pass / fail thresholds in the stress-test harness. Output is pure measurement (frame times, peak counts); callers (vitest, console) decide pass / fail.
• Drive its own ticks in perf-test mode — the auto-restart loop and per-run lifecycle live in the screen / runner that consumes notifyPerfTestRunStart(); this module only stamps run boundaries and ships snapshots.

Signals fired / Signals watched — the boss gauntlet subscribes to Sig.on('boss_kill', killHandler, 50) for the duration of each run (unsubscribed in finally); the scenario runner consumes window.__effectTriggerCounts['artifact:<id>'] as the primary effect-detection signal (zero triggers ⇒ BROKEN verdict regardless of KPI delta). No engine signals are fired by this module.

Entry points

• scenario-runner — the scenarioRunner singleton with runABSuite / runABSelected (AABB artifact proof), runTierSweep / runFullMatrix (artifact × tier × scenario sweep with baseline caching), runWeaponGauntlet (every weapon × [1,5,10,15,20] levels), runSequence (legacy linear runner), cancel, getState / getABResults / getMatrixState / getWeaponGauntletState snapshot readers, and setMissionRef / onUpdate / onMatrixUpdate / onWeaponGauntletUpdate registration.
• artifact-scenarios — ARTIFACT_AB_TESTS, ARTIFACT_AB_MAP, SCENARIO_TEMPLATES, SCENARIO_TEMPLATE_LIST, getArtifactScenario, getArtifactOverrides — bespoke A/B conditions per artifact plus the three uniform matrix templates.
• boss-gauntlet-runner — runGauntlet(mode), runSingleBoss(bossId, mode), setMissionRef, setProgressListener, getLastReport, cancel, and the window.__bossGauntlet console API mirror (run / runSingle / getReport / cancel).
• boss-gauntlet-types — the per-mode config table and the boss-run order; the shared shape between the runner and the workbench UI.
• scenario-types — the type-only module that defines ScenarioDef, PostStartAction, TestSequence, TestResult, RunnerState, MatrixState, MatrixCell, TierSweep, ArtifactMatrixResult, ScenarioTemplate, MatrixRunMetrics.
• test-sequences — ARTIFACT_SUITE_T0, ARTIFACT_SUITE_T3, WEAPON_DPS_SWEEP, STRESS_SUITE, TEST_SEQUENCES, TEST_SEQUENCE_MAP, listTestSequences().
• dev-scenarios — buildDevScenario(name), listScenarios(), parseDevOverrides(); the registry of ?dev=<name> builders (sunrise-city, elite-showcase, telegraph-test, damage-test, speed-compare, horde-peak, flame-test, empty-canvas, weapon-showcase, pillar-test, pillar-only, and the eleven art-* artifact builders).
• stress-tests — runAllStressTests(), runStressTestsWithReport(), plus the ten exported scenario functions (stressEnemySpawnStorm, stressBulletHell, stressDamageNumberFlood, stressParticleExplosionChain, stressWorldEventDensity, stressShieldHeat, stressXpOrbFlood, stressLateGameCompound, stressCollisionGrid, stressParticlesCapped); imported by tests/engine/perf/stress.test.ts and from the browser console.
• perf-test-mode — startPerfTestSession(), stopPerfTestSession(), notifyPerfTestRunStart(), isPerfTestMode(), buildPerfTestRunDef(); activated by the ?perftest URL param.

Pattern notes

• Each subsystem isolates the live world differently. The scenario runner relies on sandboxResetForTest() between runs, swaps Math.random for Mulberry32 with TEST_SEED = 20260416, and uses a manual-pump frame loop (m.testPumpFrame(fakeT) + await _wait(0)) so wave spawn cadence is frame-counted, not wall-clock-driven — that’s what makes the matrix path deterministic across machines. The boss gauntlet relies on mission reuse + sandboxResetForTest() between bosses while letting the engine’s existing rAF loop keep rendering. Stress tests skip isolation entirely and mutate engine/core/state directly, calling resetState() once per scenario.
• AABB proof shape: per artifact, A1 + A2 = baseline-no-artifact ×2, B1 + B2 = with-artifact ×2. A2 vs A1 and B2 vs B1 are determinism checks; B avg vs A avg is the effect. Verdict gating order is triggered → deterministic → deltaPct ≥ minDeltaPct. Effect-engine trigger count is the primary signal — zero triggers in B forces BROKEN regardless of KPI delta.
• Matrix sweep contrasts A/B’s bespoke conditions: uniform SWARM / SINGLE_TARGET / SURVIVAL templates with baselines pre-warmed once and cached in _baselineCache: Map<ScenarioType, MatrixRunMetrics>, then artifacts × 4 tiers run against the cached baselines. Scaling shape is derived from T0..T3 |deltaPct|: flat if both ≈0 or factor<1.1, inverted if factor<0.9, linear if T1 / T2 lie near the T0→T3 line, else exponential.
• Boss-gauntlet pass/fail is mode-specific: smoke expects the boss to survive the timeout (deliberately under-damaged), balance and survival require a kill, stress only requires no breakage. Universal gates (exception thrown, NaN bar HP) fail before the mode-specific gate runs.
• Stuck-invuln watchdog bypass ordering matters: ship._dontStuckInvuln = cfg.godMode and ship._invulnWallTime = 0 must be set BEFORE setGodMode(true); skipping seeds the watchdog with stale state and risks a false warning on later runs.
• Stress tests use shape parity with production rather than dependency injection: makeEnemy builds a synthetic enemy with the full live shape (eid, typeId, _entityType, kinematics, hp, _dying, _deathTimer, _spawnT, _spawnPopT, _spawnDur, _hitImmune, flashAmount, shapeKey, tags, etc.) so CollisionResolver, damage, and xp-orbs accept it. If the live shape grows, the helper drifts silently — and so does the gauntlet runner that this file was modeled on.
• Dev scenarios are the only subsystem that produces a real RunDefinition for normal play: baseRun deep-clones DEFAULT_RUN (JSON round-trip — safe because DEFAULT_RUN is plain data) and applies dev defaults (no fog, durable ship, 10-min timer, events unlocked), then each scenario tweaks worldKnobs / spawn / context / ship.combatStats. Artifact-test scenarios use (def.context as any).devAutoArtifacts = [{ id, tier }] as a dev-only escape hatch the bridge honors at run init.
• Perf-test mode is the only subsystem that ships to durable storage. The capture-once / attach-per-event pattern stamps device context + active flags at session start and re-attaches them to every snapshot for A/B grouping at query time. Snapshot interval is 30s; worstSpikes are sliced to top 5 to bound payload size; version and versionTag are stamped on every event so cross-build comparisons survive a mid-session BUILD_VERSION bump.
• AABB / matrix / weapon-gauntlet seeded-RNG install-and-restore is wrapped in try/finally at every call site; cancellation is cooperative (loops check _cancelled at every step boundary), but cleanup always runs.
• The boss gauntlet’s weapon-stash + _gauntletPlayerDamageDisabled flag is a belt-and-suspenders bundle: clearing weapons alone doesn’t suppress artifact-tagged damage from Personal Space or Echo Generator, so the redundant flag is necessary. Both are restored in the post-run block AND on the early-exit spawn-error path.
• Test-sequence builders synthesize artifact ids from scenario ids by stripping an art-test- prefix — a load-bearing convention shared with artifact-scenarios.ts. Renaming an artifact scenario without preserving the prefix silently grants the wrong artifact (or nothing) under ARTIFACT_SUITE_T3.
• Each subsystem owns its own tuning numbers, not the data tables: NOISY_THRESHOLD, TEST_SEED, INIT_ENEMY_COUNT, ENEMIES_PER_WAVE, mode timeoutSec / playerDpsToBoss, dev-scenario enemyCountMult / enemyDamageMult, perf-test stress curves. These are diagnostic surface knobs, not player-facing balance — kept in code, not in shared data.