Skip to main content

Hiding behind your hands won’t stop next-gen facial recognition software

face palm
As evidenced by Apple’s rumored plans to replace Touch ID with facial recognition technology for the iPhone 8, the ability of computers to seamlessly recognize faces is pretty darn impressive these days. The technology is not infallible, however, and there are still things capable of tripping it up. One example? Hands covering faces, which represents a significant challenge, due to how often a particularly animated hand gesture accidentally obscures a speaker’s face. Fortunately, computer science researchers are here to help.

What researchers from the University of Central Florida and Carnegie Mellon University have developed is a method of dealing with the so-called “facial occlusion” problem. Called Hand2Face (which admittedly sounds a little bit like that early 2000s “talk to the hand” meme), they’ve developed technology that can help improve facial recognition technology for a variety of applications — ranging from security to making machines better understand our emotions.

“Recognizing and working with facial occlusions are among the challenging problems in computer vision,” Behnaz Nojavanasghari, one of the researchers, told Digital Trends. “Hand-over-face occlusions are particularly challenging as hands and faces have similar colors and textures, and there are a wide variety of hand-over-face occlusions and gestures that can happen. To build accurate and generalizable frameworks, our models need to see large and diverse samples in the training phase. Collecting and annotating large corpus of data is time demanding, and limits many to work with smaller volumes of data, which can result in building models that do not generalize well.”


A big part of the team’s research involves building a bigger archive of hand-obscured face images for machines to learn from. This meant creating a system for identifying hands in images in the same way that present facial recognition systems identify eyes, noses, or mouths. Larger data sets can then be built up by getting the computer to automatically composite new images by taking hands from one picture and pasting them onto another. To make the synthesized images appear genuine, the computer color-corrects, scales, and orients the hands to emulate realistic images.

That’s not all, though: the method for identifying hand gestures could also be used, alongside facial expressions, to identify emotions. “In a majority of frameworks, facial occlusions are treated as noise and are discarded from analysis,” Nojavanasghari said. “However, these occlusions can convey meaningful information regarding a person’s affective state and should be used as an additional cue.”

As the need for machines to be able to read our emotions grows (consider robot caregivers, teachers, or even just smarter AI assistants like Alexa and Siri), solving problems like this is only going to become more important. You can read an academic paper describing the work here.

Editors' Recommendations

Luke Dormehl
I'm a UK-based tech writer covering Cool Tech at Digital Trends. I've also written for Fast Company, Wired, the Guardian…
Democratic lawmakers propose nationwide facial recognition ban
democrats propose facial recognition ban crowd getty

Four Democratic members of Congress unveiled a bill Thursday that would seek to ban facial recognition and other types of surveillance nationwide.

The Facial Recognition and Biometric Technology Moratorium Act of 2020 would prohibit federal agencies from investing in facial, voice, or gait recognition technology unless Congress authorizes it. The bill would also curtail the use of biometric surveillance by state and local governments, cutting federal funding if they’re in violation of the law.

Read more
ACLU files complaint against Detroit police for false facial recognition arrest
aclu files complaint against detroit police for false facial recognition test surveillance camera

The American Civil Liberties Union (ACLU) has filed a complaint against the Detroit Police Department (DPD) regarding the first known wrongful arrest in the U.S. because of facial recognition. 

The ACLU wants the DPD to stop using facial recognition technology after a Black man was wrongfully arrested earlier this year. In January, Robert Williams was arrested after facial recognition software used by Michigan State Police misidentified him as the person who robbed a watch store more than a year ago. 

Read more
Microsoft won’t sell facial recognition technology to police

Following in other tech giants’ footsteps, Microsoft announced it would not sell its facial recognition surveillance software to law enforcement. 

Microsoft President Brad Smith made the announcement Thursday during a live interview with The Washington Post. 

Read more