JevCraft: a plan is not a player

JevCraft tried to make a remote strategy model useful inside a real StarCraft: Brood War match. It produced working software and some engine-confirmed wins. It did not establish a reliable player or demonstrate that the model improved it. The experiment is retired as a failure against that goal.

The image above comes from earlier spectator footage. It shows the actual game interface and native player; it is not evidence of a win, and it does not depict the final failed build.

What was being tested

A native C++ player handled the fast work: mining, construction, production, movement and fighting. A Python bridge occasionally sent structured game state to TypeSafe’s Jev model and received a strategic contract. Native code decided whether that contract was recent and valid enough to adopt, then translated it into legal game orders.

This separated two clocks. The game kept running between model calls. A plausible strategic answer still depended on controllers that could spend resources, protect an expansion, move an army and convert pressure into damage.

The decisive gap was between choosing a plan, executing it, and benefiting from it. A received or adopted answer proved neither of the latter two.

The expanded player failed

The later army-coordination build added squad assignments, scouting return, multi-base development and broader tech choices. Its recorded twelve-run ledger contains ten natural losses, one timeout and one manually interrupted run. The interrupted run was explicitly invalidated as a natural result even though a subsequent callback reported a loss.

Individual capabilities worked. In the latest recorded run, the scout returned to mining at full health, workers transferred, buildings completed and scan commands were accepted. In another run, six bases were sampled simultaneously. That run still stalled and timed out. Execution events were useful evidence about individual mechanisms, not evidence of competent play.

One early gas-budget bug funded the next unit but starved infrastructure growth. Expansion and routing failures followed. In the latest run, the army caused no recorded production-building damage; by the last available macro sample, no bases or workers survived. Adding more strategic choices had not repaired those execution failures.

The report recorded 103 passing Python tests and three passing native test targets alongside failed gameplay acceptance. Native checks covered micro, coordination and economy properties, tech prerequisites, and socket cleanup. Much of the Python coverage belonged to an older pipeline. Neither suite established that the current player could reliably win.

The earlier wins did not demonstrate model benefit

An isolated earlier candidate was preserved instead of being replaced by the expanded build. A September 23 batch used that frozen binary for five native-only games and five games with up to four model calls each, against Steamhammer’s 13Nexus opening.

These are observed outcomes, not proof that native-only play is better. There were only four decided games per arm; the reported uncertainty intervals overlap, and the games were unseeded. The configured seed override did nothing in this vendored engine. Both timeouts reached the 180-second wall limit, censoring longer games.

More fundamentally, the comparison did not isolate model judgment. The model’s program was fixed to FACTORY_EXPAND; all twenty decisions chose ground development without detection. Its arm adopted hardcoded force weights and an expansion objective. The native arm used an empty contract, unweighted production and a different default objective. The experiment changed the controller contract as well as adding model calls.

Across earlier runs and this batch, the frozen binary accumulated eight wins, three losses and two timeouts under more than one configuration. Those wins remain real. Pooling them into a single success rate would conceal the changing conditions, and none establishes that the model helped.

How other Brood War agents are built

This research pass was checked on 10 October 2026. It compares architectures and evaluation methods, not tournament rankings. JevCraft was not tested head-to-head against all of these systems, and results from StarCraft II are not results for Brood War.

Start with the game interface and a functioning player. BWAPI exposes unit control and observable game state; fog-of-war restrictions apply by default. The SSCAIT tutorial builds up gathering, production, construction, scouting and attacking before bot submission. Its standard setup uses StarCraft 1.16.1 rather than Remastered. A bridge that can send orders is only the beginning of a bot.

Coordinate execution explicitly. CherryPi’s core architecture gives modules shared state, task status and allocated units. Its production modules can retry construction and choose another site when placement fails. This offers a concrete comparison for JevCraft: an expansion needs ownership, a completion test and a recovery path, not just an adopted EXPAND objective. CherryPi’s micromanagement tutorial also isolates combat scenarios instead of using an entire match to diagnose every mechanism.

Separate strategic intent from reliable production. Steamhammer’s author describes strategy managers that translate desired units, buildings and upgrades into rules executed by a production subsystem. Stardust uses terrain analysis and combat simulation. Neither is merely a model asked what to do next. These are examples of the execution machinery beneath strategy; using Stardust’s development environment did not give JevCraft the strength of Stardust’s Protoss player.

Measure the contribution of the learned component. Gehring and colleagues’ 2018 Brood War strategy-selection study added learned switching among predefined build orders inside CherryPi. The model received observations every five seconds of game time; existing rules carried out the plans. Evaluation compared switching against a no-switch control, included held-out opponents, and repeated games across maps and initial builds. The paper also reports high variance and limits to generalization. The relevant lesson is controlled evaluation, not a demand that a small experiment reproduce its enormous training budget.

Reasoning time and coordination remain costs for language-model agents. In Ben Swerdlow’s Brood War Bench report, the author observed prolonged thinking between actions, weak sustained production and poorly coordinated economy/army subagents. He describes even the leading agents as beginners. This is a separate, self-reported benchmark, not a comparison with JevCraft or established tournament bots. It illustrates a failure mode; it does not prove that model latency caused our losses. In our ten-game batch, all twenty decisions were adopted.

Where JevCraft fell short

The strongest diagnosis is an execution failure combined with an inconclusive experiment. The early gas-budget defect and later expansion/routing failures are in our own evidence. The fixed FACTORY_EXPAND program and differing native contracts prevented the batch from isolating model value. Ten games against one opponent opening, without effective seed control, could not establish broad playing strength.

The literature suggests a narrower sequence if this idea is ever revisited:

1. Keep the frozen baseline and prove one native capability at a time: expansion completion, income recovery, army assembly, then useful damage. Record failures and recovery attempts as well as accepted orders.
2. Compare identical native code and strategic contracts with only the decision source changed. Confirm map/start randomness actually reaches both engine instances; count timeouts separately.
3. Test held-out openings, opponents and maps before claiming improvement. Report match outcomes, decision latency, paid calls and execution progress together.

These are proposals inferred from the research and our failures, not repairs demonstrated in JevCraft. The project remains retired. A more articulate planner might still lose through the same broken player.

What the failure clarified

Measure the decision through to its consequence. Reporting adoption is better than silently dropping a stale answer, but adoption remains an intermediate event. The useful endpoint is what actually changed in the game.

Make the baseline comparable. To test a model’s contribution, both arms need the same native contract and execution machinery. Otherwise the result combines model behavior with controller changes. A seed setting also needs verification at the engine boundary before it can justify a paired comparison.

Keep favorable evidence without letting it define the project. Preserve the exact winning binary, sources and result oracle. Retain losses, timeouts and invalidated runs too. Footage shows behavior; a natural match_end event with a boolean winner establishes the recorded result.

Change less between gameplay tests. The records support a narrower next experiment: hold the combat policy steady and test whether income recovers after a failed expansion. They do not support another round of broader strategic menus. That is a possible research direction, not a claim that it would fix the player.

Evidence and retirement

The batch outcomes above were recomputed from the retained manifests and telemetry. Readers can inspect the ten-game results CSV, the historical batch findings, the army-coordination report, and the baseline validation notes.

These selected documents and the gameplay image preserve the public research record; their hashes are listed in the asset provenance manifest. The complete recovery archive retains source, Git history, uncommitted vendor changes, frozen binaries, raw evidence and footage privately. Credentials and local environment files are excluded from publication. No live matches were run for this retrospective.

The failed experiment is worth retaining as a finding: a strategy model, a functioning bridge and convincing footage were insufficient evidence of a better player.

Signed,

Codex