Your wandering gaze controls the game with Sony’s futuristic eye-tracking tech

sonys research eye tracking video games absolutely amazing sony playstation magic labs

While Sony managed to earn more than a few impressed exclamations from press, fans, and members of the gaming community with its new virtual reality headset code named “Project Morpheus,” one of its more incredible pieces of hardware sat unnoticed in a back room and had nothing to do with virtual reality.

The tech in question is a piece of unnamed prototype hardware that may or may not ever make it to retail, but it should – it literally offers a new way to look at games. Using a specially designed camera, one of Sony’s research and development teams (the group known as Magic Labs) has come up with a way for you to control the camera in a game with your eyes. It isn’t something that changes the nature of gaming in the same way that virtual reality does, but it is one of the coolest pieces of gaming tech on display at GDC.

The eye tracking technology is still a long way from being released – and there is no guarantee it will ever be – but the potential is incredible. Eye tracking is becoming a common feature of smartphones, but the scale of Sony’s device eclipses anything a smartphone currently offers. In the demo on display, Sony’s eye tracking tools (which, again, don’t technically have a name at the moment) were used in conjunction with the PlayStation 4 game Infamous: Second Son.

To help the energy blasts reach their targets, you simply look at what you want to hit.

It all starts with a quick calibration that’s as simple as looking at one dot in the upper left corner then another on the diagonally opposite side. The camera then begins to send data to a nearby laptop, which in turn takes over for the game’s camera.

It takes a few moments to orient to the new method of seeing. In a third-person game, it’s natural for your eyes to gravitate to your character that is below the center of your screen. With the eye tracking software, looking at your character therefore causes the camera to pan down. Once you compensate, though, the effects are amazing.

Beyond the ability to look around with your eyes in the same way you would normally pan the camera around with the right thumbstick, the tracking can also be far more precise. In Infamous, quickly tapping the R2 button sends a stream of energy blasts at opponents similar to firing a machine gun, but with slower projectiles. To help the energy blasts reach their targets, you simply look at what you want to hit. If you are facing two enemies, one on either side of you, just glance one direction then the other and the energy blasts will find their mark. It registers your movements as fast as your eyes move, creating a responsive camera that could alter the way we visually input games forever.  

While the demo focused on what, for all intents and purposes, is a third-person shooter (at least in the demo, which showed an early section of the game where energy blasts were among your most effective weapons), the possibilities with Sony’s eye-tracking tech are endless. Imagine playing a soccer game and, instead of trying to awkwardly pass to one particular player running in an unusual pattern, you just look at them and hit the pass button. In football, while controlling the quarterback, you could simply look at your receivers and hit pass to the one you want. In first-person shooters, you could theoretically aim faster than your thumb could move.

There are also possible uses for the tech with the Project Morpheus virtual reality headset. It’s still just theoretical right now (as far as we know), but we imagine that you may be able to one day put on the VR headset and look around the virtual world by moving your head, but also look around as you naturally would, with your eyes.

sony eye tracking Magic Labs

 The effects could go beyond gaming as well. Imagine watching a 3D telecast of a basketball game using a virtual reality headset. You could look around a limited field of view, but when the action happened, your eyes would command the video where to focus. If a shot is blocked and a fast break begins, it’s more natural to follow with your eyes than whip your head around to see the ball race down court. 

There are some drawbacks, however. For one, it requires a new camera separate from the PlayStation 4’s camera. That said, the device on display was a pure prototype, and there could always be a chance they find a way to incorporate existing cameras to use the tech via software updates. But if it were released tomorrow, you’d need to buy a new peripheral.

Another, more significant, drawback is the software’s issues with tracking eyes behind glasses. It’s hit or miss on whether or not it recognizes the movement of the eyes behind lenses, and people who wear glasses will run into issues. That could eventually be fixed. But even if you are free of spectacles, you’ll need to sit close to the camera. In order to track the movement of eyes, a camera needs to be close enough to follow the movements, potentially creating an awkward issue with positioning. The alternative would be a high-resolution camera capable of zooming in enough to make out your eyes from a distance. That’s certainly possible, but it would likely cost more than it would be worth. Still, these are all solvable issues.

Sony’s eye-tracking tech was first shown back in November 2013 as the world prepared for the arrival of the PlayStation 4. It was highlighted as part of the look at Magic Labs, the company also responsible for devices like the Move, the Eye Toy Camera, and most recently the DualShock 4. At the time, the eye tracking was little more than a gimmick – a neat gimmick – but compared to the release of the new system, a theoretical prototype was not a major focus. Now that the PlayStation 4 is on the market, however, the boundaries of gaming are ready to be pushed by devices like this one, Project Morpheus, and others we haven’t considered yet (but are likely being worked on in labs around the world).

With the Project Morpheus casting a long shadow, eye tracking is still just a sideshow, especially since there are no firm plans for it at the moment. Give it time, though. This may just be part of the future of gaming.

Product Review

Chris is the virtual co-pilot phone-obsessives need in their car

Driving while using your phone is dangerous, and often illegal. Meet Chris, the digital assistant for your car that wants to help keep your hands off your phone, and your eyes on the road.

Oculus shows off the Rift S, plans to phase out its original VR headset

Oculus plans to phase out its flagship Rift VR headset for its newly created Rift S. The Rift S made its debut this week at the 2019 Game Developers Conference and is expected to be released in spring 2019.

New Sony patent suggests a wireless PSVR headset could be on the way

Images and documents in the Japan Patent Office appear to suggest that Sony is planning a wireless version of the PlayStation VR headset. It isn't clear which system it will be used for.

These are the must-have games that every Xbox One owner needs

More than four years into its life span, Microsoft's latest console is finally coming into its own. From Cuphead to Halo 5, the best Xbox One games offer something for players of every type.

Get in the Dark Zone and blow away the competition with our The Division 2 guide

The Division 2 has three Dark Zones, specialized areas for PvE and PvP combat. Our Dark Zone guide teaches you the ropes, from the recon missions to the leveling system to the dangers and benefits of going Rogue.

Here are all the awesome award-winning indie games from IGF

Indie games are now offering experiences capable of matching -- and often surpassing -- that of big-budget blockbuster titles. The Independent Games Festival recognized the best of them on March 20.
Product Review

Acer Predator Triton 500 review

Nvidia’s new RTX 2080 Max-Q is the fastest GPU you’ll find in any laptop, but it usually comes at a steep price. Acer’s Predator Triton 500, starting at $2,500, makes it a little more affordable. But what must you sacrifice in the…

The Division 2

The latest news, guides, tips and tricks, updates, patch notes, and more so you can learn how to level up your game in The Division 2. Related Categories: The Division 2 News | Guides | Features

Get a new Nintendo Switch? You'll need to grab these accessories

The Switch is a capable console right out of the box, but it has its limitations. Thankfully, these Nintendo Switch accessories will allow you to make the most of Nintendo's latest console.

From fatalities to new characters, here's what we know about Mortal Kombat 11

Mortal Kombat 11 releases April 23 for the PlayStation 4, Xbox One, Nintendo Switch, and PC. Here is everything we know about NetherRealm's latest fighting game, including its characters.

Get a head start mastering Apex Legend's newest character Octane with our guide

Apex Legends' newest legend Octane ran so fast in the Titanfall 2 gauntlet that he blew off his legs. Now dropping into Kings Canyon with prosthetics, Octane's ability set is still centered around speed. Here's how to play as Octane.

The Division 2 offers nothing but a funhouse mirror of America

Tom Clancy's The Division 2 improves on the design shortcomings of the original game in several different ways, but its version of Washington D.C. is completely removed from reality.

World of Warcraft's allied races will make you want to start a new character

The Hord and Alliance are seeking new allies in their struggle for control of Azeroth. Rather you pledge your allegiance to the Horde or Alliance, we've got a guide to help you unlock every allied race that's coming in Battle for Azeroth.

How the Google Stadia could lead to a new era of multi-GPU gaming

Google's Stadia could use more than one graphics card to deliver the high-performance visuals it's promised. If that leads to better developer support for multi-GPUs, could that mean gaming with two or more graphics cards could finally be…