# Plan 007 findings: seeded batches and a native-only baseline

Run on 2026-09-23 at branch `improve-2026-09-23`.

## Budget

Operator instruction: implement all plans. The plan's default budget applied:
5 games per arm, at most 20 paid Jev requests, UI on (the only proven harness
mode).

## Seeding: STOP, the ini path does nothing

- BWAPI loads `seed_override` from the `[starcraft]` section
  (`vendor/.../BWAPI/Source/BWAPI/GameMenu.cpp:20`) into
  `GameImpl::seedOverride`.
- Nothing in the vendored OpenBW BWAPI reads `seedOverride` afterwards. A
  search of `3rdparty/openbw` finds only the declaration, the load, and a
  commented example in `bwapi.ini.example`. The setting is a no-op here.
- `Game::setRandomSeed(uint32_t)` exists in the public API (`Game.h:1749`,
  sets OpenBW's `lcg_rand_state`). In a two-client LAN lockstep game it would
  have to be called identically by both modules on the same frame to avoid a
  desync. The frozen stepwise binary can't be changed to call it, and start
  locations are assigned before `onStart`, so it wouldn't control them anyway.
- Next spike, if paired comparisons are needed: find where OpenBW's LAN
  handshake chooses the game seed (the sync code under `openbw/openbw/sync*.h`)
  and add an env-controlled override there, as a new patch in
  `patches/stardust/`.

Consequence: replicates are independent, unseeded games. That's fine for
estimating a win rate per arm, but a native-vs-Jev difference can't be paired
game by game.

## Measurement

Frozen stepwise binary (SHA-256 `935e5dce…d119`), Steamhammer `13Nexus`,
program fixed to `FACTORY_EXPAND`, fast simulation, UI on, 180 s wall limit.

| Arm | Wins / decided | Wilson 95% | No result (180 s timeout) | Paid requests |
|---|---|---|---|---|
| native (`JEV_STRATEGY_MAX_REQUESTS=0`) | 4 / 4 | 0.51–1.00 | 1 | 0 |
| jev (≤4 requests/game) | 2 / 4 | 0.15–0.85 | 1 | 20 |

Per game (`python3 tools/summarize_batch.py runtime/evidence/batch-2026-09-23-*`):

| Run | Result | Frame |
|---|---|---:|
| native-r01 | win | 43,186 |
| native-r02 | win | 53,168 |
| native-r03 | no result (timeout) | — |
| native-r04 | win | 24,524 |
| native-r05 | win | 54,005 |
| jev-r01 | loss | 26,570 |
| jev-r02 | win | 62,654 |
| jev-r03 | no result (timeout) | — |
| jev-r04 | loss | 18,727 |
| jev-r05 | win | 32,956 |

All 20 Jev decisions were adopted (0 rejected). Every one chose
`FACTORY_EXPAND` (fixed by the harness) + development `GROUND` + no
detection.

## What this does and doesn't show

- **The arms don't compare model judgment.** Jev's answers were constant, so
  the only real difference is that the Jev arm adopts the `FACTORY_EXPAND`
  contract's hardcoded force weights (Marine 0.3, Vulture 1, Tank 1, objective
  `EXPAND`). The native arm runs with an empty contract: unweighted
  production, default objective `HARASS`.
- **No evidence that the model helps in this setup.** Native-only did at least
  as well (4/4 against 2/4), but the intervals overlap heavily and n=4 decided
  games per arm can't separate them. Don't claim either arm is better.
- **The frozen binary's combined record is 8 wins, 3 losses.** That's 2 wins
  and 1 loss before this batch plus 6 wins and 2 losses across both arms, over two
  configurations and unseeded. It is not a single win rate.
- **Timeouts:** 2 of 10 games hit the 180 s wall limit, so long games are
  censored. Raise the limit before the next batch, or games that would have
  gone long get dropped from the sample.

## Recommended next steps

1. Let Jev actually choose: run the Jev arm without
   `JEV_STRATEGY_FIXED_PROGRAM` so program choice is live. That requires
   plan 005's deferred-plan path, which exists only in `src/`, not in the
   frozen binary.
2. Isolate the contract effect for free: a native arm whose default contract
   equals `FACTORY_EXPAND` shows whether the force weights, rather than the
   model, account for the gap.
3. Increase n (≥15 decided games per arm) and the timeout (e.g. 300 s) before
   drawing any conclusion. Native-only games cost nothing.
4. The engine seed override (see Seeding above) would enable paired
   comparisons and cut the required n substantially.