We’re all so worried about AI making our games, but is the real danger that AI will start playing our games? Xbox has been testing out a new AI Gaming Copilot as “the ultimate gaming sidekick” to help players who get stuck or even want to improve their skills without having to go look up a guide. Now, I am perhaps more open to the idea of AI in games than most people, but this strikes a little too close to home. Across all my years in the industry, I have written thousands of guides so I have a natural aversion to the idea of an AI invalidating all that time and effort.
I wanted to be fair, though, and give Xbox Gaming Copilot AI a chance to change my mind now that it is available on Xbox PC. To do this, I decided to play a game and let the AI completely take the wheel and make every decision for me. Even being generous since it is still in beta, I can’t believe how poorly it went.
AI, take the wheel
The only decision I had to make to properly test out Gaming Copilot was what game to have it play. I knew it had to be something turn-based or puzzle-based to accommodate the need to stop playing, ask it what to do, and then return to the game for every action. I ran through a list of games and decided on Monster Train 2. If you’re unfamiliar, this is a deckbuilding, auto-battler roguelike where you place monsters on a train to fight incoming waves of enemies. Even if you have no idea what any of that means, it will make sense as I describe it.
I booted up the game and set things up to be as simple as possible; I chose the first two clans and knocked the difficulty modifiers down to the base level. I got Copilot set up and chose the “wise” voice option with the hopes that it might translate into its actual intelligence.
The first choices I passed along to the AI. Which of the two heroes should I pick and which of the two artifacts should I choose before the first battle? It was reluctant at first to make a choice for me, but with a little force I managed to get it to commit to one or the other. While it didn’t pick the hero I would’ve, it did pick the better of the two artifacts — applying Pyregel 5 on strike instead of randomizing the cost of all cards except my champion between 1 and 3 and drawing +2 cards.
That’s about the last good decision Copilot made.
Copilot had me turn on the trail on the first fight that makes enemies appear on every floor on the first wave for extra gold and had me place one unit on each of the three floors, with my hero unit in the middle. Not optimal, but fine so far.
On my first turn, Copilot wanted me to play a spell card that attacks a single unit, but claimed it would hit all enemies on that floor. I didn’t question it and played along as close as I could.
Next turn, I drew a card put into my deck by an enemy that only serves to take up a slot in my hand. Coplilot told me to play that card by name, insisting it would “weaken enemies on the middle floor.” It also told me to play a buff card called Just Cause on the enemies, saying it would damage them. Later, it decided that Just Cause now actually did 35 damage and caused stagger. Neither could be further from the truth.
Just to be clear, every card in my deck has its effects written on it. Copilot is supposed to read my screen to understand what’s going on and use that to provide useful information, but it is clearly skipping the “reading” part. It is looking at my screen; it knows how much energy I have and what cards I have to play, but now what any of them do despite being written clear as day. It not only made up what cards did, but frequently changed its mind about what they did.
I could go on with more examples about how it told me to take a left at the fork to get a Banished unit when that reward was on the right, or how it said the next trial would cause Pyre damage (which results in a loss if I run out), or told me to attack enemies that weren’t present, but I think I’ve illustrated the point well enough.
But maybe a game with so much strategy and interpretation was too complex for an AI. I mean, it failed to even read the cards, but I was willing to give it just one more shot with something more objective.
Return to Monkey Island is a classic point-and-click title with occasionally obscure but static puzzles. It isn’t on the level of games from the ’90s where you needed to rub every object you had together to find a solution, but it can be a little tricky. Still, there’s a clear solution to everything and I personally wrote what amounts to a near-complete walkthrough of the game, so there’s no excuse for Gaming Copilot to steer me wrong.
Are you surprised to learn that it did?
The first “puzzle” in the prologue of the game is to get coins for a scurvy dog, aka a gross hot dog, which is outright said to be in the bathroom. But the door is locked, so I asked my “wise” Copilot how to get the key. Much more than in Monster Train 2, it was very reluctant to give me any kind of straight answer. It repeated that I should “look around, talk to people, and solve puzzles” to try and find it. Eventually, it spat out that the key was inside a pirate’s coat. Now, there is a sleeping pirate on the balcony on this screen, but that’s not where the key is. The key is inside the shop selling scurvy dogs.
At this point, I’d seen enough. Xbox Gaming Copilot AI isn’t just unfit to play a game for me, but I wouldn’t trust it to tell me what the Start button does. It’s knowledge of what I’m playing seems to start and end at the title of the game, seemingly not able to even parse information currently on the screen, let alone find answers elsewhere. At least for now, all of us guide writers should rest easy knowing that Xbox Gaming Copilot AI isn’t just bad, but straight-up wrong most of the time.
