Skip to main content

Facebook’s ‘droidlet’ A.I. could take speech recognition to a whole new level

A distorted Facebook logo.
Digital Trends / Digital Trends

Between Facebook’s core social network product and apps like Messenger and WhatsApp, Mark Zuckerberg’s tech giant has undeniably changed the way people communicate. Could it next help change the way we communicate with robots?

In 2021, the idea of being able to communicate with artificial intelligence through natural language is nowhere near as science fiction as it once was. Whether it’s Amazon’s Echo voice assistant or the voice bots we interact with when you phone your bank, A.I. today means that machines can do a pretty good job of understanding what humans are asking for, without the human in question having to do much to modify the way that they’re speaking.

However, robots aren’t quite so user-friendly. Partly because most robots are still used in industrial or lab-based settings, where there’s not quite the same requirement for them to be accessible to everyday users, robot interactions remain more opaque in their operation. Breakthroughs in locomotion, gripper dexterity, and image recognition mean that robots can now be trained to navigate different environments and interact with objects. But asking a robot to, for instance, “pick up the blue tube next to the fuzzy chair that Bob is sitting in” is still far beyond the scope of today’s robot platforms.

At least, that’s the popular conception.

Researchers in Facebook’s A.I. Research laboratories have been working to change this — and, in the process, they’re developing technology that won’t just imbue robot platforms with smart assistant-style natural language abilities but, so it claims, far exceed current standards.

Facebook and the neural semantic parser

“We don’t want to speculate on the tech in Siri or Echo, but natural language understanding for home assistants is still often based on rules or narrowly-defined neural models, and often only can handle specific user ‘intents’ — for example, ‘send message,’” Mary Williamson and Arthur Szlam of Facebook A.I. (FAIR) told Digital Trends via email.

Facebook’s open-source technology, called “droidlet,” uses what it calls a “neural semantic parser” to provide robots with a better way to understand the real world.

“For example, for the command, ‘move to the red chair,’ the droidlet agent parses the command, and then searches its memory for any previously observed object it might have detected that has been tagged by its object-property model as being ‘red’ and ‘chair,’” the researchers said. “In order to create this object detection, it [uses] a data stream from the robot’s sensors, perhaps mediated by a library like [open-source robot software framework] ROS. Using this memory, it builds a move command directing the body to the location of the chair. That move command is [then] sent to a lower level interface like ROS that executes the move on the robot.”

The droidlet system comprises multiple pieces of core technology including a dialog system, perceptual or vision modules, a structured memory system for storing and retrieving “memories,” and more. Facebook says that its open-source technology can be used for controlling multiple real-world robots such as LoCoBot and Hello-Robot, as well as for robot simulators like Habitat. If the company is successful with the project, it promises that it will allow researchers to more easily build robot agents able to carry out complex tasks in both real life and in simulated environments like Minecraft.

Infographic of facets of Droidlet machine learning.
Image used with permission by copyright holder

Big tech’s robot love

Facebook is far from alone in the world of big tech giants investing in robotics. In some cases, the moves are more immediately commercially applicable than in others. Amazon, for instance, has its own Amazon Robotics division, previously known as Kiva Systems when it was bought by the online retail giant in 2012 for $775 million. Amazon’s robots fit neatly into its mission statement because they can be used in the company’s fulfillment centers to help dispatch orders. At present, several hundred thousand robots are used in this capacity.

Google, on the other hand, has also long maintained an interest in robots — although its most famous acquisition, robot dog Spot-makers Boston Dynamics, was acquired in late 2013 and then offloaded to Japan’s SoftBank Group just a few years later in mid-2017. With no buying or selling price ever confirmed, it’s impossible to know just what Google got out of the deal, but it certainly wasn’t its own commercial (or even non-commercial) robot.

With Facebook being a company that has dipped its toes into hardware even less frequently than Google (its Portal video-calling devices being a rare exception), how does the world’s biggest social media giant stand to gain from its latest robot fixation?

“While our work is focused purely on A.I. research, we believe the long-term applications are plentiful,” Williamson and Szlam said. “[We] see social interactions in natural language with learned, embodied agents — robots, smart wearable devices such as AR glasses, et cetera — for various kinds of future assistants as an interesting area for Facebook in general.”

That’s not exactly confirming that Facebook robots are coming soon. While many Facebook products over the years started life as bits of research at FAIR, the research lab does not necessarily carry out its work with an immediate eye on commercialization. However, this work is certainly — in Facebook terminology — a “poke” that indicates interest in ramping up its focus on all things robot. It remains to be seen whether that becomes anything timeline-worthy.

Editors' Recommendations

Luke Dormehl
I'm a UK-based tech writer covering Cool Tech at Digital Trends. I've also written for Fast Company, Wired, the Guardian…
CES 2022 shows the very convoluted future of video game tech
Asus ROG Flow Z13 gaming laptop.

A lot has changed about the technology we use to play video games. For decades, players needed to have a PC, console, or portable device (whether it be a Game Boy or iPhone) to game. Nowadays, the lines between all three of those categories have blurred. Devices like the Nintendo Switch have sparked a portable console revolution, while cloud gaming is making companies rethink what devices gamers need to play AAA titles.

That philosophical shift has been on full display throughout CES 2022. Manufacturers showed up in full force to unveil incredibly powerful devices that further break the established gaming mold. While there are plenty to ooh and ahh at, but this year's show highlights just how convoluted gaming tech has become.

Read more
Digital Trends’ Tech For Change CES 2022 Awards
CES 2022 Tech for Change Awards Post Feature

CES is more than just a neon-drenched show-and-tell session for the world's biggest tech manufacturers. More and more, it’s also a place where companies showcase innovations that could truly make the world a better place -- and at CES 2022, this type of tech was on full display. We saw everything from water-conserving showers to affordable electric trucks. But of all the world-changing tech on display this year, these four innovations impressed us the most:

Whill Model F
No matter how advanced or capable they might be, most powered wheelchairs available today suffer from the same fundamental flaw: They aren’t easy to transport when they’re not in use.

Read more
The best smart lights of CES 2022
GE Cync lights set up in gaming room.

CES 2022 is rapidly drawing to a close, and in the midst of it all were a few new smart lights that caught our eye. Let's see what's new from the show and what you might want to include in your next smart lighting upgrade.
GE Cync

GE's lighting brand Cync unloaded 11 new smart bulbs, covering all the form factors you could ask for, including filaments, candelabras, and globes. It pledged to support Matter in the future, too, which is good news for playing along with your other smart home devices. Expect to see these on store shelves at Lowe's, Best Buy, Target, and Amazon in March, with price points starting at $12. You can read up more about Cync's other announcements, including smart thermostats and security cameras.

Read more