Your wandering gaze controls the game with Sony’s futuristic eye-tracking tech

sonys research eye tracking video games absolutely amazing sony playstation magic labs

While Sony managed to earn more than a few impressed exclamations from press, fans, and members of the gaming community with its new virtual reality headset code named “Project Morpheus,” one of its more incredible pieces of hardware sat unnoticed in a back room and had nothing to do with virtual reality.

The tech in question is a piece of unnamed prototype hardware that may or may not ever make it to retail, but it should – it literally offers a new way to look at games. Using a specially designed camera, one of Sony’s research and development teams (the group known as Magic Labs) has come up with a way for you to control the camera in a game with your eyes. It isn’t something that changes the nature of gaming in the same way that virtual reality does, but it is one of the coolest pieces of gaming tech on display at GDC.

The eye tracking technology is still a long way from being released – and there is no guarantee it will ever be – but the potential is incredible. Eye tracking is becoming a common feature of smartphones, but the scale of Sony’s device eclipses anything a smartphone currently offers. In the demo on display, Sony’s eye tracking tools (which, again, don’t technically have a name at the moment) were used in conjunction with the PlayStation 4 game Infamous: Second Son.

To help the energy blasts reach their targets, you simply look at what you want to hit.

It all starts with a quick calibration that’s as simple as looking at one dot in the upper left corner then another on the diagonally opposite side. The camera then begins to send data to a nearby laptop, which in turn takes over for the game’s camera.

It takes a few moments to orient to the new method of seeing. In a third-person game, it’s natural for your eyes to gravitate to your character that is below the center of your screen. With the eye tracking software, looking at your character therefore causes the camera to pan down. Once you compensate, though, the effects are amazing.

Beyond the ability to look around with your eyes in the same way you would normally pan the camera around with the right thumbstick, the tracking can also be far more precise. In Infamous, quickly tapping the R2 button sends a stream of energy blasts at opponents similar to firing a machine gun, but with slower projectiles. To help the energy blasts reach their targets, you simply look at what you want to hit. If you are facing two enemies, one on either side of you, just glance one direction then the other and the energy blasts will find their mark. It registers your movements as fast as your eyes move, creating a responsive camera that could alter the way we visually input games forever.  

While the demo focused on what, for all intents and purposes, is a third-person shooter (at least in the demo, which showed an early section of the game where energy blasts were among your most effective weapons), the possibilities with Sony’s eye-tracking tech are endless. Imagine playing a soccer game and, instead of trying to awkwardly pass to one particular player running in an unusual pattern, you just look at them and hit the pass button. In football, while controlling the quarterback, you could simply look at your receivers and hit pass to the one you want. In first-person shooters, you could theoretically aim faster than your thumb could move.

There are also possible uses for the tech with the Project Morpheus virtual reality headset. It’s still just theoretical right now (as far as we know), but we imagine that you may be able to one day put on the VR headset and look around the virtual world by moving your head, but also look around as you naturally would, with your eyes.

sony eye tracking Magic Labs

 The effects could go beyond gaming as well. Imagine watching a 3D telecast of a basketball game using a virtual reality headset. You could look around a limited field of view, but when the action happened, your eyes would command the video where to focus. If a shot is blocked and a fast break begins, it’s more natural to follow with your eyes than whip your head around to see the ball race down court. 

There are some drawbacks, however. For one, it requires a new camera separate from the PlayStation 4’s camera. That said, the device on display was a pure prototype, and there could always be a chance they find a way to incorporate existing cameras to use the tech via software updates. But if it were released tomorrow, you’d need to buy a new peripheral.

Another, more significant, drawback is the software’s issues with tracking eyes behind glasses. It’s hit or miss on whether or not it recognizes the movement of the eyes behind lenses, and people who wear glasses will run into issues. That could eventually be fixed. But even if you are free of spectacles, you’ll need to sit close to the camera. In order to track the movement of eyes, a camera needs to be close enough to follow the movements, potentially creating an awkward issue with positioning. The alternative would be a high-resolution camera capable of zooming in enough to make out your eyes from a distance. That’s certainly possible, but it would likely cost more than it would be worth. Still, these are all solvable issues.

Sony’s eye-tracking tech was first shown back in November 2013 as the world prepared for the arrival of the PlayStation 4. It was highlighted as part of the look at Magic Labs, the company also responsible for devices like the Move, the Eye Toy Camera, and most recently the DualShock 4. At the time, the eye tracking was little more than a gimmick – a neat gimmick – but compared to the release of the new system, a theoretical prototype was not a major focus. Now that the PlayStation 4 is on the market, however, the boundaries of gaming are ready to be pushed by devices like this one, Project Morpheus, and others we haven’t considered yet (but are likely being worked on in labs around the world).

With the Project Morpheus casting a long shadow, eye tracking is still just a sideshow, especially since there are no firm plans for it at the moment. Give it time, though. This may just be part of the future of gaming.

Gaming

‘Spyro Reignited Trilogy’ proves nostalgia only goes so far

Spyro Reignited Trilogy is a faithful and beatiful remaster of the PlayStation classics, but its dedication to retaining the original experience ultimately hinders the experience. Spyro was great in the late-'90s, but it hasn't aged well.
Gaming

Every rumor about the PS5, including a new game from Luminous Productions

PlayStation 5 rumors have been circulating for over a year now but there's still plenty we don't know. Here's everything you need to know about the PS5, including rumors about its release, specs, and games.
Gaming

These are the best Xbox One games available right now

More than four years into its life span, Microsoft's latest console is finally coming into its own. From 'Cuphead' to 'Halo 5,' the best Xbox One games offer something for everyone.
Deals

The best cheap-but-awesome PS4 game deals under $20

The PlayStation 4 has hit its stride in recent years and is now more affordable than ever. If you have a PS4 or are thinking of buying one, we’ve collected some must-have games. The best part? Each of these is just $20 or less.
Gaming

The best PS3 games of all time

Choosing the right PlayStation 3 game can be a conundrum, especially when there are nearly 1,500 titles to choose from. Thankfully, we've rounded up the best games to have ever made it to the platform.
Gaming

Here's where Xur is and what he has for wares this week in 'Destiny 2: Forsaken'

The weekly vendor in Destiny 2: Forsaken always brings Exotic weapons and armor, some of the toughest loot to find in the game. Here's everything you need to know to track down Xur: Where he is, when he shows up, and what he's stocking.
Features

From NES to PlayStation, retro gaming is booming. But will it last?

Everyone craves the good old days, and companies like Nintendo and Sony are cashing in with their mini versions of beloved retro consoles. But is this fad built to last or will our fixation with pixelated graphics come and go like mom…
Gaming

Microsoft is reportedly planning an Xbox One without a disc drive for 2019

Microsoft is reportedly planning to release a new Xbox One model without a disc drive in 2019. The console will be significantly cheaper than other Xbox One systems, including the Xbox One S.
Gaming

15 tips for keeping your vault-dwellers alive in ‘Fallout Shelter’

The wasteland can be an unfriendly place, if you don't know what you're doing. Here are 15 tips that will help your vault thrive in Fallout Shelter, including information on questing.
Gaming

Stock up on Switch accessories this week with these awesome Black Friday deals

Black Friday is right around the corner, and it's the perfect time to stock up on accessories for your Nintendo Switch. Here are the best Switch accessory deals on Black Friday, from controllers to cases and more.
Product Review

Make friends. To survive the wasteland of 'Fallout 76,' you're gonna need 'em

Fallout 76 brings the nuclear wasteland online and invites you to hop in with your friends. It’s fun, and a new direction for the Fallout franchise, but it’s not without its faults.
Gaming

‘Final Fantasy VII Remake’ director sets record straight on game’s progress

Final Fantasy VII Remake director Tetsuya Nomura clarified some information about the game's progress. There were earlier reports that the RPG was put on hold in favor of Kingdom Hearts III, but that is apparently not the case.
Gaming

‘Blockchain gaming’ startup gets $16 million in Series A funding

A gaming startup, Mythical Games, has raised $16 million in Series A funding to work on their concept of blockchain gaming. The aim is for the startup to develop games for the PC, console, and mobile markets.
Gaming

‘Fortnite’ named Ultimate Game of the Year at the Golden Joystick awards

This year's Golden Joysticks, the gaming awards handed out by GamesRadar, have been announced and Fortnite has been named the Ultimate Game of the Year. It beat out Red Dead Redemption for best game, but Red Dead was named critics' choice.