Facebook's CherryPi suffered defeat in 'StarCraft,' but a win for its AI team

The StarCraft AI competition took place during AIIDE (Artificial Intelligence and Interactive Digital Entertainment) 2017 at the Snowbird Ski Resort in Utah from October 5 to October 9. The annual conference focuses on artificial intelligence and interactive entertainment, pooling talents from the commercial and research communities. The highlighted topic covered artificial intelligence beyond its use in gaming.

The StarCraft AI competition specifically used the original StarCraft PC game and its expansion, Brood War. Its purpose was to evaluate the state of artificial intelligence, and how it handles real-time strategy games. This genre poses a challenge because the player must scan the field, manage resources, investigate unknown environments, and respond quickly to threats. Chess doesn’t offer that strategic depth.

Although StarCraft: Brood War has an integrated artificial opponent, it has access to the game’s back end. It doesn’t view the screen as a human does — it’s not a detached entity that must think on its own. Thus, for an external “bot” to experience the game like a human, a special interface called BWAPI was created for bots to interact with the PC game.

“BWAPI only reveals the visible parts of the game state to AI modules by default,” the Github notes state. “Information on units that have gone back into the fog of war is denied to the AI. This enables programmers to write competitive non-cheating AIs that must plan and operate under partial information conditions.”

In Facebook’s defense, the ZZZKBot is a mean opponent. It just participated in the 2017 IEEE SIG StarCraft AI competition in August and came out winning 1,790 games out of 2,374. At the end of the local tournament in Utah, the ZZZKBot came out on top again with 2,465 wins out of 2,966 games. Written by Chris Coxe, it first made an appearance during the AIIDE 2015 conference.

But given that Facebook’s CherryPi bot managed sixth place out of 28 opponents in its first swing, the company did not exactly suffer total defeat. Facebook’s AI team behind CherryPi are composed of eight individuals who published a dataset last week drawing on early data collected from CherryPi’s gameplay. The data includes 496 million player actions and millions of captured frames.

All bots were ranked by their final winning percentage of one-on-one games. They could not cheat and could not take advantage of in-game glitches. Each session lasted up to 60 minutes with fog of war enabled, which covers unexplored areas of the field. Bots were penalized for slow computations.

Out of the 28 competing bots, 15 were created and submitted by independent developers. Even more, all five bots ranking higher than CherryPI were created by independent developers. The only other non-independent AI developer to reach the top 10 was Stanford University and its Arrakhammer bot.