Can LLM-written bots beat the best human-written bots?

Not yet. Against the top human-written bots Stardust, PurpleWave, and BananaBrain, the best model (GPT-6 Astra) wins only 7% of its games.

What predicts a model's score?

The strongest predictor is how much the model practiced against the 7 strongest roster bots during the hour. Models that started playing those bots early — even while their own bots still lost to them — scored higher.

How do scores compare to other benchmarks?

StarSkirmish still tracks Vals IOI and WeirdML v3, but barely correlates with the Artificial Analysis Intelligence Index (ρ 0.26) and has drifted from Terminal-Bench 4.0, suggesting heavily benchmarked evals have become tuning targets while less-targeted hard coding tests still move together.

StarSkirmish Bench — LLMs write StarCraft bots in a 1-hour coding showdown

StarSkirmish Bench is a benchmark where LLMs write bots for StarCraft: Brood War. Each LLM gets one hour of wall-clock time to write a Protoss bot in C++ (using BWAPI 4.4.0 on OpenBW), and the resulting bot is evaluated against a fixed roster of competitive human-written bots and demo bots. v0.2 runs flagship models from OpenAI, Anthropic, Google and Meta in each lab's first-party CLI harness at max reasoning effort, and scores each model by the average win rate of its 5 runs. GPT-6 Astra placed first, but the strongest human-written bots (Stardust, PurpleWave, BananaBrain) still beat every model decisively — the best model wins only 7% of its games against them.

Key Features

1-hour agentic coding challenge

Each LLM gets one hour of wall-clock time to write a working Protoss bot in C++, starting from a skeleton bot that only mines, with source code of three demo bots plus the FAP combat simulator and BWEM map analyzer. The bot must keep compiling; whatever is in the bot folder when the hour ends is the

Win-rate scoring against a fixed roster

A model's score (0–100) is the average win rate of its 5 final bots over tournament games against 13 roster bots (10 competitive human-written bots and 3 demo bots), 6 games per pairing on each of three maps. v0.2 replaced v0.1's Elo ratings with plain win rates so scores stay comparable across mode

First-party CLI harnesses at max reasoning

v0.2 runs each lab's models in its own first-party CLI harness (Claude Code, Codex, Antigravity CLI, Muse Code) at the highest reasoning setting each offers, rather than one neutral third-party harness as in v0.1.

Tiered practice opponents with feedback

During the hour, the LLM can compile its bot, play practice games, and read each game's summary, transcript, and its own bot's log. Practice opponents are tiered D through S (demo bots, mid-ranked human bots, BananaBrain/Locutus, and Stardust/PurpleWave). Practice against the strongest bots is the s

Cost-vs-score analysis

The bench publishes each model's score against the average API cost of one 1-hour run on a log scale, highlighting value tradeoffs (e.g., GPT-6.1 Sol scoring second at just over a tenth of GPT-6 Astra's cost per run).

Run-to-run variance reporting

Each model's score is the mean of 5 independent runs, with the spread (lowest to highest run) and standard deviation published, exposing differences such as Gemini 4 Argon's 18-point gap between best and worst runs.

How It Works

Game and format

Games are Protoss versus Protoss on three ladder maps (Heartbreak Ridge, Benzene, Destination) using BWAPI 4.4.0 on OpenBW. A game ends when one side's buildings are all destroyed; games hitting the 60-minute cap (86,400 frames) are decided by kills + razings score as on the BASIL ladder.

Agent setup

The LLM agent starts with a skeleton bot that only mines, the source of three demo bots, and two libraries (FAP and BWEM). It has tools to compile, play practice games against tiered opponents, and read game summaries, transcripts, and logs. Web search and web fetch are disabled with no other intern

Tournament and scoring

Each model's 5 final bots play each of 13 roster bots 6 times per map on fresh seeds; LLM bots don't play each other. All bots play the same fixed roster, so a new model's bots can be scored in parallel without replaying anyone else's games, and the bar stays constant over time.

Pros & Cons

Pros

Tests genuinely long-horizon agentic coding: a real game engine, real compile-play-iterate loops, and a hard deadline with no submit step
Fixed-roster win-rate scoring keeps the bar constant, making scores comparable across model generations without replaying other LLM bots
Transparent methodology: harness versions, CLI versions, reasoning effort settings, context compaction behavior, and tournament structure are all documented
Scores are averages of 5 independent runs with variance reported, not single-run results
Cost-per-run is published alongside scores, enabling value comparisons like GPT-6.1 Sol's second-place score at a tenth of GPT-6 Astra's cost

Who It's For

Best for

AI labs and researchers who want a hard, hard-to-game test of long-horizon reasoning and agentic coding
Practitioners choosing between frontier models who care about value per agentic coding run, not just raw scores
StarCraft: Brood War bot enthusiasts interested in how LLM-written bots compare to established human-written ones

Not ideal for

Teams needing full StarCraft coverage — only Protoss v Protoss on three maps is tested
Anyone looking for a general-purpose coding benchmark aligned with popular evals like Terminal-Bench or the Artificial Analysis Intelligence

Use Cases

Evaluating how far frontier LLMs have come at long-horizon agentic coding

Comparing labs' flagship models under realistic, first-party tooling rather than a neutral harness

Tracking whether general model progress correlates with hard, less-targeted coding benchmarks (e.g., Vals IOI, WeirdML v3)

Studying agentic behaviors like when models choose to practice against stronger opponents

Measuring the cost-effectiveness of a model's agentic coding performance per run

FAQ

What is StarSkirmish Bench?

A benchmark where LLMs write bots for StarCraft: Brood War. Each LLM gets one hour to write a Protoss bot in C++, which is then evaluated against a fixed roster of competitive human-written bots and demo bots. Scores are average win rates (0–100) of each model's 5 runs.

How does a model earn its score?

Each model gets 5 independent runs. Every final bot plays each of 13 roster bots (10 competitive human-written bots and 3 demo bots) 6 times per map on three maps — Heartbreak Ridge, Benzene, and Destination — with LLM bots not playing each other. A model's Bench score is the average win rate of its 5 bots.

Which harnesses and settings are used in v0.2?

Each lab's models run in their first-party CLI harness: Claude Code (Claude models), Codex (GPT models), Antigravity CLI (Gemini 4 Argon), and Muse Code (Muse Wonder Full), at the highest reasoning effort each supports (Gemini 4 Argon at high, all others at max). Context is compacted at roughly 250k tokens.

Can the models use the internet during the run?

StarSkirmish Bench