Death of the mouse: How eye-tracking technology could save the PC

Eye Tracking

In the last few years, touch control has revolutionized the way we interact with mobile devices. The technology has been so popular on smartphones that Apple used its proven touch approach to reinvent the dead tablet market with the iPad. Thanks to the blooming growth of these devices, touch is taking off in a big way, taking on new form factors and posing a potential threat to our oldest friend: the PC.

With all of these motion-controlled interfaces for video game systems and touch interfaces for mobile devices, the PC with its keyboard and mouse, just feels, well, old. The keyboard is still the fastest and best way to enter large amounts of data and to author written content, but the mouse and touchpad are a step removed from the natural, direct feeling one gets when using the Wii, Xbox Kinect, or a touch tablet. Tobii hopes to rectify this imbalance.

Last Friday, I met up with Barbara Barclay, North American manager of Tobii Technologies (a Swedish company) to try out a completely new type of user interface built for consumer desktops and laptops. In a small office building in Manhattan, New York, she let me try out one of only 20 prototype Lenovo laptops, which each have built-in infrared sensors that track eye movement so precisely and quickly that it makes even the best mouse interfaces feel antiquated.

Here’s how it works

Lenovo-Tobii eye control PCBefore we began the demo, Barbara explained the technology. Tobii’s eye control works a bit like the Xbox Kinect (or a reverse Wii), but on a much closer scale. As you sit in front of the laptop, a row of two synced infrared sensors located under the screen scan your eyes. They do this about 30 to 40 times per second, examining the size and angle of your pupil, the glint in each of your eyes, and the distance between you and the laptop. Together, the two sensors create a stereoscopic 3D image of your eye for the computer to examine. Based on the angle and glint of your eye, Tobii’s technology calculates precisely which part of the screen you are looking at. It can even tell when you look away or close your eyes. To save power, the demo unit on hand darkened its screen when we looked away.

After explaining how it works, Barbara calibrated the Lenovo-Tobii eye control PC for her eyes. The calibration process takes a few seconds. Basically, you look at a series of three to nine dots on the screen, which lets the computer know where your eyes are looking. Nintendo has used similar calibration tests on its Wii Motion Plus controllers and Wii Fit balance board software. The calibration is painless and shouldn’t have to be done very often. After calibration, the laptop will be able to save your “eye profile” and know how to calibrate when a familiar user logs on.

It actually works…really well

Next, we ran through several hand-crafted demos that show off some different use scenarios for eye control. Even after I tried it out myself, I had a hard time believing the demonstration wasn’t an elaborate ruse. However, after a few moments, I began to believe.

Almost instantly, the computer began to pick up my eye movements and respond to whatever I looked at. When I looked at an item, it would highlight itself and come to the forefront. When I looked at a map, it knew exactly which area I was staring at, right down to the pixel. Much like how a novice with the Wii will wave the Wii Remote wildly, my first instinct was to move my entire head as I looked at different items on the screen. This worked well enough, but after a few moments, I learned that Tobii’s technology could pick up the subtlest of eye movements without the aid of my head moving. Somehow it could tell when I moved my eyes a half inch to the left or right or casually looked up or down, even a hair.

Lenovo-Tobii eye control PC

The fluidity of the experience reminded me of the first time I used the iPod Touch, and how natural it felt to swipe and touch precisely where I wanted. Before the Touch and iPhone, most touchscreens used resistive touch technology, which required you to actually press down on the screen. These screens demanded a stylus (pen-like device) to achieve precision, but Apple changed the game with its more natural interface that let you directly use your fingers. Tobii’s eye control technology is as direct as any touch interface. It feels like touch from afar.

Uses for eye control

The first portion of the demo (which you can watch below) simply shows where your eyes are looking on the screen. Looking at your own eyes isn’t particularly fun, but it shows you how fast the system reacts when you move or blink. However, after the intro screen and calibration, we got into some different use scenarios.

Reading: Out of all of the uses for eye control, reading demonstrates its value more than anything else. Everyone has their own technique and style for reading on a laptop or touch device. Personally, I tend to keep my text toward the top of the screen. Sometimes I use the mouse to highlight things I’ve already read and use the direction buttons to scroll down. Tobii’s eye control instantly makes all of these customized reading styles irrelevant. More natural than book reading, text automatically scrolls up, down, left, or right for you as your eyes pan around the screen. It’s amazing. Confused about a word? Well, if you stare at one word long enough its definition will pop up.

Playing media: Another demo Barbara showed me was a simple media player. A row of pictures and album covers  fill the bottom of the screen. Glancing at one of them highlights the choice and looking upward plays the music or maximizes the picture so you could get a better look. Done listening or viewing? Simply look at another item in the list. And when you looked at the arrows on the left or right for a second or so, the next page of results appear.

Zooming and panning: Eye control doesn’t mean there is no use for the keyboard. In a Google Maps-like demo, you can pan and zoom by looking and pressing/holding a button, which works well. A single button is assigned to zoom and another button is assigned to pan. To zoom, you simply look at what you wish to focus on and push the zoom button. Once zoomed, holding another button and looking left, right, up, or down lets you pan around your zoomed image. There are likely even more intuitive ways to perform complex tasks like this.

Multitasking: At CES this year, I complimented the BlackBerry PlayBook for its easy swiping method to switch between applications. With a WebOS-like interface (or perhaps WebOS itself) and Tobii’s eye control, multitasking between apps is as natural as looking to the left or right of the screen. Using Windows 7, which isn’t at all optimized for anything other than a mouse, Barbara swapped between windows by looking at them and pressing a button. She also moved a mouse pointer icon around the screen with ease.

Gaming: The last demo we played was a simple Asteroids-like game. Your mission is to protect the Earth from a barrage of doomsday-sized comets and asteroids headed your way. Looking at an asteroid triggers a laser that blows it up. There are a ton of touch-type games like this, which pit your reflexes against the computer, but eye control takes the speed and intensity of these games up a notch. I’m incredibly excited to see what kind of games can be made using the speed of the eye.

These are only a few of many new ways eye control would let you interact with a PC desktop or laptop. When you start thinking about how this technology could interact with voice recognition, the possibilities seem endless. In a few years, Minority Report may look dated.

It’s like Kinect for PCs

Lenovo-Tobii eye control PC

Microsoft has said that it will eventually release Kinect-like motion technology for PCs. Well, I hate to break it to them, but Tobii has done a lot of the hard work. Moving Kinect to a PC would mean shifting the focus from the body to the eyes and face, something Tobii has achieved with remarkable precision.

I’ve written a lot about Microsoft and the many challenges it faces with Windows 8. Currently, the company has a failing Phone platform and no tablet strategy. To forge ahead, Microsoft needs to generate excitement around its bread and butter, which is still traditional, keyboarded PCs. Tobii’s eye control is exactly the kind of innovative interface Microsoft could build a comprehensive new experience around. It could simplify the complexities of the Windows desktop OS while bridging some of the gaps between the PC and mobile touch platforms. If implemented properly, technology like this could reignite some buzz around the laptop market, especially combined with many of the ideas Microsoft has demonstrated on Windows Phone and Xbox Kinect.

Barbara informed me that Microsoft is already a client of Tobii’s for some of its larger research units, which the Redmond giant uses to study the effectiveness of its own application layouts and interfaces by tracking the eye movements of potential customers as they experience a new design. Microsoft, I’m looking at you. If you don’t try something like this, someone else will.

Regardless of who it is, one thing is clear: Tobii needs a strong partner that realizes the potential of its technology.

Eye control is coming, hopefully

Like any good technology, Tobii’s eye control is only as useful as the software developers who write for it. The company is in talks with a number of hardware, software, and platform makers to work toward implementing its technology in PCs as soon as two years from now, but it will be a tough road ahead. The technology still needs to be smaller, use less battery, and cost less. Without the vision of Apple, touch tablet devices stagnated in limbo for a decade. Lets hope that one of these platform makers recognizes the potential in eye control. The only loser here is the mouse, and I’m sorry my friend, I love ya, but your days are numbered.

Emerging Tech

Awesome Tech You Can’t Buy Yet: 1-handed drone control, a pot that stirs itself

Check out our roundup of the best new crowdfunding projects and product announcements that hit the web this week. You may not be able to buy this stuff yet, but it sure is fun to gawk!
Home Theater

What is MHL, exactly, and how does it work with your TV?

There are more ways to mirror your smartphone or tablet to your TV than you might think. Check out our rundown of MHL for everything you need to know about the wired protocol and its myriad uses.

Xbox One adds mouse and keyboard support, levels playing field with PC

The latest system software update for Xbox One has added mouse and keyboard support. The support is limited to Warframe and Fortnite at the moment, with more games coming in November and December.
Product Review

Rylo is the 360 camera that finally makes 360 video useful

The Rylo puts a new spin on 360 video by focusing on flat, fixed-frame output. It can’t quite replace your action camera or camcorder, but it shows a glimmer of what the future of consumer video could be.

These Windows 10 keyboard shortcuts will update your OG Windows skills

Windows 10 has many new features, and they come flanked with useful new keyboard shortcuts. Check out some of the new Windows 10 keyboard shortcuts to improve your user experience.

Protecting your PDF with a password isn't difficult. Just follow these steps

If you need to learn how to password protect a PDF, you have come to the right place. This guide will walk you through the process of protecting your documents step-by-step, whether you're running a MacOS or Windows machine.

iPhone users are finding themselves randomly locked out of their Apple ID

According to posts on Reddit and Twitter, it looks like users on Reddit and Twitter having some issues with their Apple accounts. Specifically, it seems as though users are getting randomly locked out of their Apple IDs.

Don't know what to do with all your old DVDs? Here's how to convert them to MP4

Given today's rapid technological advancements, physical discs are quickly becoming a thing of the past. Check out our guide on how to convert a DVD to MP4, so you can ditch discs for digital files.

Here’s how to install Windows on a Chromebook

If you want to push the functionality of your new Chromebook to another level, and Linux isn't really your deal, you can try installing Windows on a Chromebook. Here's how to do so, just in case you're looking to nab some Windows-only…

Edit portraits with A.I. and adjust focus in the new ON1 Photo RAW 2019 editor

ON1 Photo RAW 2019 now has a dedicated tab for portraits that automatically recognizes faces to help with retouching. The update also brings a new focus stacking tool, enhancements to layers, and improvements to local adjustments.

Your MacBook can live in the lap of luxury with this leather case

Though there are several cases which we think are best for covering up MacBooks, Twelve South's Journal case is one of the newest available, providing luxurious leather coverage for your Apple laptop.

Here's our head-to-head comparison of Pandora and Spotify

Which music streaming platform is best for you? We pit Spotify versus Pandora, two mighty streaming services with on-demand music and massive catalogs, comparing every facet of the two services to help you decide which is best.

15-inch MacBook Pro gets more powerful with new AMD Vega GPUs

Confirming Apple's quiet October announcement, new configurations for the top-range 15-inch Apple MacBook laptop are now available, coming complete with AMD Pro Vega 16 or Pro Vega 20 graphics cards on board.
Emerging Tech

Intel’s new ‘neural network on a stick’ aims to unchain A.I. from the internet

To kick off its first developer conference in Beijing, Intel unveiled the second generation of its Neural Compute Stick -- a device that promises to democratize the development of computer vision A.I. applications.