At the Neural Information Processing Conference in Montreal, Canada, Nvidia researchers demonstrated that they could use the power of artificial intelligence to render synthetic, yet realistic, scenes with details and textures. Researchers claim that the work is still in its early stages, and it is unclear when the technology will be released to consumers, but there is big potential for Nvidia’s artificial intelligence-driven rendering in the gaming space.
“This work is about a new way of rendering computer graphics using neural networks,” Nvidia’s Vice President of Applied Deep Learning Bryan Catanzaro said in a conference call. Basically, researchers wanted to know how they can apply A.I. to make computer graphics better, and the solution is to apply machine learning to real-world videos in order to render new graphics.
“We’ve built a system that takes high-level representations of the physical world — basically taking a video sketch and convert that into a rendered scene,” Catanzaro said. “The model understands high-level information of objects in the real world, and then elaborate those to add texture and lighting information. The goal is to be able to synthesize new scenes with graphics.”
Machine learning is used to analyze existing videos and Nvidia would apply computer vision techniques to add labels to objects and their properties. This means that A.I. would be able to recognize an urban cityscape and understand what objects are trees, cars, or building, for example.
This technology is derived from existing research, like that from the University of California, Berkeley, according to Nvidia. The company has shown off other A.I.-based rendering techniques in the past, including one that would remove noise from an image.
Researchers were able to achieve real-time rendering on a Tensor Core GPU, but for the conference, Nvidia demonstrated the technology on its Titan V card. “Though you can do this on any processor, the real-time aspect does require a lot of A.I. throughput,” Catanzaro explained, noting that the Tensor Core GPU is important.
By being able to create content and adding them to virtual worlds, the gaming market could greatly benefit from the work created by this research. Developers, for example, could “remaster” old games by re-rendering old titles to add high-definition visuals, or they could add new levels to existing games with little effort.
As a basic example of how this would work, users can take a video of themselves, upload it to a game, and the rendering will be able to create highly personalized avatars for use in the game. Nvidia has open-sourced the code from its research right now, but Catanzaro cautions that the early work in this field is more suited for computer scientists than game developers.
In a separate demo, Nvidia showcased that it could analyze dance moves — like from the popular music video Gangnam Style — and transfer those moves to another person using the same computer vision techniques. “We analyze the code of the person, and that becomes our sketch,” Catanzaro said. “And the model renders the target person given that sketch.”
Nvidia cautions that this is still an early-stage technology, but you can also upgrade the graphics from an existing game and train a model on real-world imagery to re-render old games to make them look better. Because the technology requires the computer to analyze objects that are known in the real world, Catanzaro cautions that it won’t work on fantastic stuff, like rendering Santa’s elves. In theory, he admits, you can train the computer to render elves, but you need to create physical elves and capture images for the computer to learn.
Like the ray tracing technology that was introduced on Nvidia’s recent RTX series graphics cards for consumers, the company said that this A.I.-based rendering is used in a hybrid way and isn’t meant to replace traditional rendering techniques. Instead, A.I.-based rendering is meant to coexist and be used with traditional graphics rendering engines.
Right now, it’s unclear when this technology will hit the gaming market. It could take as little as a couple years, Catanzaro optimistically speculated. Combined with ray tracing, A.I.-based scene rendering will deliver quality visuals in game titles that are rendered in real time.
