Skip to main content

Facebook’s ‘droidlet’ A.I. could take speech recognition to a whole new level

A distorted Facebook logo.
Digital Trends / Digital Trends

Between Facebook’s core social network product and apps like Messenger and WhatsApp, Mark Zuckerberg’s tech giant has undeniably changed the way people communicate. Could it next help change the way we communicate with robots?

In 2021, the idea of being able to communicate with artificial intelligence through natural language is nowhere near as science fiction as it once was. Whether it’s Amazon’s Echo voice assistant or the voice bots we interact with when you phone your bank, A.I. today means that machines can do a pretty good job of understanding what humans are asking for, without the human in question having to do much to modify the way that they’re speaking.

However, robots aren’t quite so user-friendly. Partly because most robots are still used in industrial or lab-based settings, where there’s not quite the same requirement for them to be accessible to everyday users, robot interactions remain more opaque in their operation. Breakthroughs in locomotion, gripper dexterity, and image recognition mean that robots can now be trained to navigate different environments and interact with objects. But asking a robot to, for instance, “pick up the blue tube next to the fuzzy chair that Bob is sitting in” is still far beyond the scope of today’s robot platforms.

At least, that’s the popular conception.

Researchers in Facebook’s A.I. Research laboratories have been working to change this — and, in the process, they’re developing technology that won’t just imbue robot platforms with smart assistant-style natural language abilities but, so it claims, far exceed current standards.

Facebook and the neural semantic parser

“We don’t want to speculate on the tech in Siri or Echo, but natural language understanding for home assistants is still often based on rules or narrowly-defined neural models, and often only can handle specific user ‘intents’ — for example, ‘send message,’” Mary Williamson and Arthur Szlam of Facebook A.I. (FAIR) told Digital Trends via email.

Facebook’s open-source technology, called “droidlet,” uses what it calls a “neural semantic parser” to provide robots with a better way to understand the real world.

 

“For example, for the command, ‘move to the red chair,’ the droidlet agent parses the command, and then searches its memory for any previously observed object it might have detected that has been tagged by its object-property model as being ‘red’ and ‘chair,’” the researchers said. “In order to create this object detection, it [uses] a data stream from the robot’s sensors, perhaps mediated by a library like [open-source robot software framework] ROS. Using this memory, it builds a move command directing the body to the location of the chair. That move command is [then] sent to a lower level interface like ROS that executes the move on the robot.”

The droidlet system comprises multiple pieces of core technology including a dialog system, perceptual or vision modules, a structured memory system for storing and retrieving “memories,” and more. Facebook says that its open-source technology can be used for controlling multiple real-world robots such as LoCoBot and Hello-Robot, as well as for robot simulators like Habitat. If the company is successful with the project, it promises that it will allow researchers to more easily build robot agents able to carry out complex tasks in both real life and in simulated environments like Minecraft.

Infographic of facets of Droidlet machine learning.
Image used with permission by copyright holder

Big tech’s robot love

Facebook is far from alone in the world of big tech giants investing in robotics. In some cases, the moves are more immediately commercially applicable than in others. Amazon, for instance, has its own Amazon Robotics division, previously known as Kiva Systems when it was bought by the online retail giant in 2012 for $775 million. Amazon’s robots fit neatly into its mission statement because they can be used in the company’s fulfillment centers to help dispatch orders. At present, several hundred thousand robots are used in this capacity.

Google, on the other hand, has also long maintained an interest in robots — although its most famous acquisition, robot dog Spot-makers Boston Dynamics, was acquired in late 2013 and then offloaded to Japan’s SoftBank Group just a few years later in mid-2017. With no buying or selling price ever confirmed, it’s impossible to know just what Google got out of the deal, but it certainly wasn’t its own commercial (or even non-commercial) robot.

With Facebook being a company that has dipped its toes into hardware even less frequently than Google (its Portal video-calling devices being a rare exception), how does the world’s biggest social media giant stand to gain from its latest robot fixation?

“While our work is focused purely on A.I. research, we believe the long-term applications are plentiful,” Williamson and Szlam said. “[We] see social interactions in natural language with learned, embodied agents — robots, smart wearable devices such as AR glasses, et cetera — for various kinds of future assistants as an interesting area for Facebook in general.”

That’s not exactly confirming that Facebook robots are coming soon. While many Facebook products over the years started life as bits of research at FAIR, the research lab does not necessarily carry out its work with an immediate eye on commercialization. However, this work is certainly — in Facebook terminology — a “poke” that indicates interest in ramping up its focus on all things robot. It remains to be seen whether that becomes anything timeline-worthy.

Luke Dormehl
I'm a UK-based tech writer covering Cool Tech at Digital Trends. I've also written for Fast Company, Wired, the Guardian…
The craziest and most creative product concepts from CES 2022
Razer Project Sophia render.

CES wouldn't be CES without its usual assortment of wonderful concept designs that will never actually manifest as real-world products, and this year's event didn't disappoint.

It's always fun to see what shows up, with the creative efforts offering a glimpse at how tech outfits have been splashing their R&D cash as they search for the next big hit.

Read more
Rainbow Six Extraction will turn you into a better squad mate
A squad attacks an alien in Rainbow Six Extraction.

My first Rainbow Six Extraction mission didn’t quite go as planned.

During a demo event, I was paired up with two strangers who would become my close squadmates for nearly four hours. Having completed a solo tutorial with little trouble, I went into our inaugural mission thinking I’d carry my team to victory as a party of one. One swarm of slimy aliens later, I was out of action, cocooned in a sickly yellow shell. My poor teammates had to abort the mission, toss my immobilized body on their shoulders, and extract me to safety. I was quite literally dead weight.

Read more
Everything LG announced for the smart home at CES 2022
LG room interior with sofa and drawer with mockup poster near window.

LG has almost always had a presence in the smart home space, but the company made that clearer than ever before at CES 2022 with a flurry of new smart home devices aimed at improving people's lives. From improving our air quality and turning us into the perfect cooks to adding more settings to washers and dryers, the LG ThinQ app plays a larger role than ever before.
PuriCare AeroTower

The company aims to not only improve your air quality, but also your comfort with the LG PuriCare AeroTower. Not your ordinary air purifier, the PuriCare AeroTower is also a fan and a heater. The model supports three airflow modes (with 10 levels of intensity each): Wide, Diffusion, and Ddirect. Depending on the model, you can add heat (up to 86 degrees Fahrenheit) to any airflow mode. The unit uses multistage True HEPA filters to effectively capture most particles floating around a room. It also uses ultraviolet-C light to help combat bacteria. For those who want to check on the air quality, they can look on the device or check out the LG ThinQ app on their phone.
InstaView Double Oven Range

Read more