Skip to main content

How generative AI will create games with ‘broader, bigger, and deeper worlds’

Of all the use cases for generative AI out there, I can’t think of one more significant than video games. Sure, we’ve seen people create simple games from GPT-4 — but surely, I assumed a technology this powerful was being discussed at the higher levels of game development too.

To get an idea of how big of a shift this could be, I wanted to talk to someone who actually understands how games are made on a technical level. Marc Whitten, Senior Vice President and General Manager of Unity Create, is surely one such person. He’s particularly excited about how AI could transform game development, and we spoke about how the tools that could enable that revolution are already making their way to creators.

Faster create time

Ziva Face Trainer in Unity.

Games take a huge amount of time and effort to develop, but most of that time is dedicated to creating all of the content for the game. Whitten says that if you look at a common 300-person AAA studio, somewhere around 80% of them are dedicated to creating content. AI can speed up that process drastically.

Whitten provided a clear example of that: Ziva Face Trainer. Ziva is a company Unity acquired in early 2022, and it has been working on its Face Trainer tool for a little over two years. It takes a model, trains it on a large set of emotions and movements, and generates something usable.

How much time does this save? Whitten says high-end rigging of a character can take a team of four to six artists four to six months: “Candidly, [that’s] why the state-of-the-art quality of characters has not actually progressed that much in the last ten years or so.”

Senua’s Saga: Hellblade II – The Game Awards 2019 – Announce Trailer (In-Engine)

With Ziva Face Trainer, developers “give it a mesh and we train that mesh against a large set of data… so you get back in five minutes a rig model that allows you to then run it in real-time.” Ziva tech is being used a lot, too. It’s behind the suit deformation in Spider-Man: Miles Morales, as well as the Troll in the Senua’s Saga: Hellblade 2 trailer. You’ve probably seen it in a few movies and TV shows even — Captain Marvel, John Wick 3, and Game of Thrones are on the list.

That shouldn’t come as a surprise. Machine learning and procedural techniques (such as tools like SpeedTree) aren’t exactly new in the world of game development. It’s true that more research into AI models can lead to even more efficient creation pipelines, but we’re seeing a shift with generative AI. We’re talking about large language models (LLMs) like GPT-4 and diffusion models like Midjourney, and they can radically change the games we see.

Changing the game

Whitten says the hope with AI is to make games “ten to the third better,” which means games that are ten times faster, ten times easier, and ten times cheaper to develop. The result of that isn’t a flood of the same games we have, though. Whitten believes the results of that are “broader, bigger, deeper worlds.”

I asked for an example, and Whitten pondered what Skyrim would look like if it had a generative AI model behind it. We’ve all heard the “arrow to the knee” meme from the game, but Whitten imagined a game where that throwaway line meant something more.

“Well, what if each of those guards actually had a Myers-Briggs-type chart? A little bit of a backstory and frankly, a backstory that could have been impacted by that. What has happened with the character along the way? And then an AI model to generate what would be a rational response coming out of that, given all of those particular events.”

We’re seeing some effort there with games like The Portopia Serial Murder Case, which, bluntly, haven’t made the best case for AI in games. It’s not hard to see the potential, though, especially in larger games with NPCs that don’t have set quests or exhaustive dialogue.

A player converses with an NPC in The Portopia Serial Murder Case.

There’s a lot of potential in sandbox-style games, as well. Whitten imagined a GTA-style game where you “go into the pawn shop and recruit the person behind the behind the desk and, you know, with maybe the game creator never even thinking about that as a possibility because of something else that happened in the game.” Whitten also thought about Scribblenauts, except in a world where you could truly make anything and assign it any properties.

The problem right now is getting that to actually work, as evidenced by The Portopia Serial Murder Case. Whitten was one of the founding members of the Xbox team at Microsoft, and he helped lead the Kinect push. About Kinect, Whitten said: “I would tell everybody it works amazing if I’m sitting next to you.” You needed to prompt it in a specific way, and if you deviated, it wouldn’t work.

That’s the big problem that’s faced AI as a whole, with smart assistants like Alexa only operating within a narrow range. LLMs change that dynamic and allow for any prompt, and that’s what’s exciting about creating deeper game worlds. There’s still a road to get there, though.

“If you put the tool out there … [creators will] hit whatever the boundaries are and say, ‘Well, that’s not fun.’ But then they’re going to actually go find the space that no one’s even thinking about,” Whitten said.

With more tools rolling out, we could see some early experiments with AI within the next year. We already have in some cases, such as the wildly popular AI Dungeon 2. But to make this sort of immersive world possible at scale, you need a middleman. And for Unity, that middleman is Barracuda.

The Barracuda

An image from Unity's Book of the Dead.

Unity includes a neural network inference library called Barracuda. As Whitten explains, “It’s an inference engine that allows you to drive either diffusion or other forms of generative content at runtime on the device without hitting the cloud and at a highly performant pace.”

Oh yeah, performance. As much as we like to talk about AI can change content forever, there’s a massive computational cost (there’s a reason it took tens of thousands of GPUs to build ChatGPT). Barracuda allows those models to run on your CPU or GPU so you don’t have to go out to the cloud, which, for the record, would be a huge money-sink for developers.

Unity is working on more features for Barracuda, and Whitten says the “interest back from the game creator community has been extraordinarily high.” It’s the key that makes generative AI possible in game development and design, especially without requiring any specific hardware.

Whitten says the team wants to start “building techniques that allow creators to start really targeting a large and core part of their game design, not ‘Oh, this is going to really diminish my audience if I design for it.'” Unreal Engine, for its part, has a similar tool (the aptly-named NeuralNetworkInference tool, or NNI).

These libraries, when met with large generative AI models and an acceleration in content development, can lead to an “explosion of creativity,” according to Whitten. And that’s something to get excited about for the future of games.

This article is part of ReSpec – an ongoing biweekly column that includes discussions, advice, and in-depth reporting on the tech behind PC gaming.

Editors' Recommendations

Jacob Roach
Senior Staff Writer, Computing
Jacob Roach is a writer covering computing and gaming at Digital Trends. After realizing Crysis wouldn't run on a laptop, he…
AI is coming for your PC games, but you should be excited, not worried
how ai can change destory pc gaming games respec featured

The tech community has been oversaturated with AI this past week, from ChatGPT to Google Bard, but not without reason. We see fads like NFTs and web3 come and go, but AI is here to stay -- even in your PC games.

It's not all doom and gloom, though. AI and machine learning has already proven itself wildly useful in PC gaming, and it has far-reaching implications for how games are made and experienced. I'm not trying to fit a square peg into a round hole here -- and if you stick with me, you'll see why.
How it's being used now

Read more
I built a couch gaming PC that puts the PS5 to shame — and you can too
A PC sitting next to a PS5 on a coffee table.

The PlayStation 5 is back in stock, and if you've been eagerly waiting to jump into the next generation of gaming, now seems like the time to strike. I'm here to sway you away from a console, though, because you can build a PC for around the same price that puts the PS5 to shame.

PC building is getting more expensive, but prices on the best graphics cards are dropping. With some clever shopping and bit of elbow grease, a gaming PC can deliver better performance and higher-quality visuals without costing much more than Sony's console. Here's the build you need.
Meet the PS5 killer

Read more
Gaming monitors have been lying to us, and it’s time they stopped
A gaming monitor playing Forza Horizon 5.

Gaming monitors are lying to us, and they have been for many years. Informed buyers know the tricks that brands play to sell the best gaming monitors, and they've learned to navigate the deceptive marketing. But these ploys persist, and 2023 is the year when monitors need to get a little more transparent.

Some of the key areas where gaming monitors mislead buyers have been running rampant for years, while others are fairly new. As we start a new year and look onto next-gen displays, consider this buying advice for picking up your next gaming display, as well as a call to arms to demand manufacturers do better.

Read more