Need a ruler for your tablet? Qeexo conjures virtual tools with natural gestures

“Everybody uses pinch and zoom, because it’s easy to remember,” Sang Won Lee, CEO of a gesture control company called Qeexo told Digital Trends. “The reason we don’t use three- or four-finger gestures is because we can’t remember what they do — but you can probably hold a computer mouse with your eyes closed, right?”

Absolutely. Try making that shape with your hand right now, and we’re pretty sure you’ll succeed. Now imagine using that same gesture to summon up a virtual mouse on the screen of your tablet; or in the future, an interactive tabletop, complete with left and right buttons, and that always-helpful scroll wheel. Such natural movements could transform the way we interact with touchscreens, especially large ones.

Qeexo can make those kinds of gestures possible right now, and we got a chance to see its impressive technology in action.

Machine learning

Anyone who has used a modern Huawei phone may be familiar with Qeexo’s first product. It’s called Finger Sense, but Huawei changed the name to Knuckle Sense for use in its Mate S, Mate 8, the P9, and other phones. The concept is simple. You use your knuckle instead of a fingertip to activate various features, such as taking a screenshot or opening apps. Underneath Finger Sense is a versatile machine learning engine, around which Qeexo has built Touch Tools and Impact Sense: two tools that go way beyond using just our knuckle.

“We don’t want to make people learn a whole lot of new gestures, we want to make gestures that people are already familiar with.”

The mouse gesture is only the start for Touch Tools. It can also generate a pen, pencil, or paint brush, then move on to erasers of all sizes, a magnifying glass, a tape measure, a ruler, and even a camera for taking quick screenshots. The ability to conjure helpful tools on demand with simple gestures would certainly make our lives easier and limit the number of accessories and third-party apps we buy to meet our needs. Qeexo puts all the functionality of old-school real-world tools quite literally at the tips of our fingers.

For the last couple of years, Qeexo has refined its machine learning code to teach our devices how we, as humans, actually wield and use tools. The company has demonstrated the tech in other products such as FingerAngle along the way, and it’s the real star of the show. You simply teach the machines the way you personally hold a mouse or a pen, so that it always interprets the gesture correctly. It doesn’t just guess what you’re doing based on what hundreds of other people do, plus it will adapt to changes in the way you “make” the mouse, camera, or any other gesture over time. Adding this kind of intelligence increases the likelihood that the machine will correctly guess which type of gesture you made to provide you with the tool you want.

Beyond mobile

We watched Qeexo’s Touch Tools work on an iPad, and it looked great. It was a super slick demo that made me grin, just like all the best tech innovations should. However, it gets really exciting when you start to consider its use outside of a mobile device. The gestures seen here can be adapted to any shape and size, so could be made into a knob or dial, ready for use on a touchscreen inside a car.

For instance, a large touchscreen dominates the Tesla Model S’s dashboard, and although it looks amazing, it’s a problem to use on the move because you have to look at the display to find settings and make adjustments. That’s not a good idea when you’re moving along. Now imagine reaching out and grabbing a virtual dial to adjust the volume of the stereo or turn down the heat. For Touch Tools, this is simple; and for the driver, it’s a potential life saver.

“We don’t want to make people learn a whole lot of new gestures, we want to make gestures that people are already familiar with,” Sang continued. But how would we differentiate between dials, and understand we’re grabbing the right one without looking? “The button for the volume in my car is smaller than the one for the AC, so you’d make a different gesture, but what we can also do is link the gesture to haptic feedback, and as soon as you turn the dial you’ll feel that the engine has activated the right tool.”

Because conflicts could arise between similarities in gestures, the company will work with app developers to integrate only the most relevant tools for the situation. We don’t think anyone will need a magnifying glass in the car, for example. Qeexo’s keen to avoid gesture overload, which is one of the problems it wants to solve.

Impact Force

Impact Force (which sounds like the title of another Dirty Harry movie, but isn’t) is a little different to Touch Tools, but uses the same learning engine. It’s designed as an alternative to pressure-sensitive gesture systems, like 3D Touch on the iPhone 6S and the Apple Watch.

“People are struggling to trigger force sense functionality. They can’t tell the difference between a long press and a hard press,” according to Sang, “because we don’t have a pressure sensor on the end of our finger.” Qeexo sees value in the concept of a third selection process, but also wants to solve it on a software level, making it more accessible to manufacturers that don’t only produce high-end, expensive smartphones.

Impact Force reacts with a sharp impact on the display, and the machine learning engine will be able to understand the difference between a tap, a long press, and a hard/sharp press. It’s an alternative to changing out hardware to measure pressure, but there is a difference between the two approaches. “We can do most of the things a pressure sensitive screen does, but not continuous pressure sensing, such as that used for Huawei’s weight measurement trick.”

Coming soon

Why haven’t we seen this approach to gesture controls before now?

“It took us two and a half years to perfect,” said Sang. “We had to do a lot of optimization work based on feedback from customers, such as bringing down the power consumption, and get the CPU usage under a certain limit.”

Geexo Gesture
Andy Boxall/Digital Trends
Andy Boxall/Digital Trends

Now it’s ready for the big time. Qeexo expects Impact Force to make an appearance on a smartphone soon. The company hinted that it may happen after the summer. For Touch Tools, Qeexo is working with an OEM partner, that Sang wouldn’t name for obvious business reasons, and is hoping to have it inside a device we can buy before the end of 2016.

Notice we said, ‘device’ rather than a phone or tablet? According to Sang, “It’s bigger than a tablet. Not a 55-inch TV or something, but between a typical tablet and TV. It will be targeting both regular people and businesses.”

Once these devices reach consumers, the times of ignoring gesture controls because we can’t remember what to do may be close to an end.

Mobile

Dozens of bite-sized features add up to a feast in Android 9.0 Pie

Google’s latest version of Android is here, and it’s called Android 9 Pie. We’ve been using it for the past few months on a Google Pixel to see how this flavor stacks up.
Mobile

Master your iPhone XS and iPhone XS Max with our favorite tips and tricks

If you're one of the lucky folks that managed to snag an iPhone XS or XS Max, you may be wondering how to use your new phone in a post-home button world. Here are a few of our favorite iPhone XS tips and tricks to get you started.
Computing

Pain in the wrists? Type in comfort with one of these great ergonomic keyboards

Long typing sessions can leave anyone's wrists aching, but if you have one of the best ergonomic keyboards, that doesn't have to be the case. Our list of favorites will support good typing posture while being comfortable to use.
Mobile

Not sure about updating to iOS 12? Here are five reasons why you should

If you’re on the fence about whether to install iOS 12 or not, allow us to explain why the update is worthwhile. Here are five of our favorite features from Apple’s mobile platform.
Mobile

Huawei is not-so-subtly trolling Friday’s iPhone launch

Apple launches the iPhone XS range to the public on Friday, but Huawei is out in force to remind the public what they could be missing out on (Hint: It's the Mate 20 Pro) by choosing Apple's latest smartphone.
Mobile

How to buy the iPhone XS, iPhone XS Max, and iPhone XR in the U.K.

The new iPhone range is here, and it consists of three models: The iPhone XS, the iPhone XS Max, and the iPhone XR. You can buy the iPhone XS and XS Max in the United Kingdom now, so here's our guide on where to buy one.
Mobile

Audio company Bragi is suing OnePlus over the word 'dash'

Despite taking steps to change to "Warp Charge," OnePlus is being sued by audio company Bragi over the phone manufacturer's continued use of the word "dash" in the Dash Charging used in OnePlus phones.
Mobile

Need a do-over? Here's how to factory reset an iPhone, from XS on down

Resetting an iPhone can alleviate all sorts of software woes, and wipe away personal data should you sell your device or give it to someone else. Here's how to factory reset an iPhone from within iOS or iTunes.
Mobile

The best weather apps for Android will keep you dry no matter where you go

You may not be able to change the weather, but you can at least be prepared for it. Check out our guide to the best weather apps for Android, so you'll always know what to expect when you step out the front door.
Mobile

Android 9.0 Pie is finally rolling out to the OnePlus 6

Android 9.0 Pie has been released. But is your phone getting Android 9.0 Pie, and if so, when? We've done the hard work and asked every device manufacturer to see when their devices would be getting the update.
Mobile

Keep the iPhone XS display crack-free with these screen protectors

Apple might have proclaimed the iPhone XS's glass as being its most durable ever, but that's not going to stop you from wincing if you drop your phone. Stay protected with the best iPhone XS screen protectors.
Mobile

Apple iPhone XS Max vs. Huawei P20 Pro: Clash of the titans

Anyone seeking a great new smartphone with plenty of money to spend has two amazing options, but which is better for you? We pit the Apple iPhone XS Max vs. Huawei P20 Pro in various categories to help you choose.
Product Review

Don't let the bigger iPhones woo you away: The XS is still a masterpiece

Apple’s next smartphone is here -- the iPhone XS. We think it’s the perfect size for an iPhone, and it manages to impress with astounding performance, and sizable camera improvements.
Product Review

With its epic screen, Apple's iPhone XS Max is a phone you can live inside

The iPhone XS Max is here. Should you get the massive 6.5-inch iPhone from Apple? Or should you pick the smaller iPhone XS? We’ve been putting the Max through its paces to find out in our review.