Skip to main content

Drones are no longer crash-test dummies thanks to MIT’s new VR training platform

Autonomous Drone Racing in FlightGoggles

To better train drones and reduce the risk of damage to itself and its surroundings, MIT engineers developed a training platform called “Flight Goggles” based on virtual reality. This enables a fast-flying drone to train within a virtual environment while speeding through empty physical space. Given the nature of VR, these drones can now safely train for any environment and condition. 

Recommended Videos

Without Flight Goggles, drone training typically includes a large enclosed area with nets to catch “careening” vehicles and physical props including doors and windows. If they crash, that’s an added expense to the project due lost time, repairs, or a complete drone replacement. This type of training is ideal for slow-moving drones designed to scan an environment, not fast-moving models. 

“The moment you want to do high-throughput computing and go fast, even the slightest changes you make to its environment will cause the drone to crash,” says Sertac Karaman, associate professor of aeronautics and astronautics at MIT. “You can’t learn in that environment. If you want to push boundaries on how fast you can go and compute, you need some sort of virtual-reality environment.” 

To develop Flight Goggles, the team began with a “hangar-like gymnasium” lined with motion-capture cameras mounted on the walls to track the drone’s movement through physical space. This data is inserted into an image rendering program that generates a photorealistic virtual environment based on the drone’s position and perspective. The program then sends that combined data back to the drone.  

According to Karaman, the drone’s camera isn’t on, and instead “hallucinates” as it “sees” one environment while speeding through another, processing that visual feed at 90 frames per second. The drone used to test Flight Goggles was based on a 3D-printed nylon and carbon fiber frame, a custom-built circuit board, an embedded “supercomputer,” an inertial measurement unit, and a camera. 

For the initial test, the team created a virtual living room with a window twice the size of the drone. Flying at five miles per hour, the vehicle darted through the virtual window 361 times and “crashed” only three times. All throughout this test, the team tweaked its navigation algorithm so the drone could “learn on the fly” and avoid virtual walls. 

Of course, had the team used props instead of VR in this experiment, three repairs or complete drone replacements would be in order. But with Flight Goggles, the drone could “crash” thousands of times and the training would continue without costly repairs and downtime. 

But you can’t have a VR training session without testing the drone in a real-world scenario. The team built the same window within the facility, and then turned on the drone’s on-board camera. The result: It zipped through the physical window 119 times and crashed/required human intervention six times. 

While that doesn’t sound entirely successful, remember that the fast-flying drone learned to fly in virtual space not to mention zooming through the opening at 5 miles per hour. Karaman believes Flight Goggles could even safely train drones to fly around humans. 

Kevin Parrish
Former Digital Trends Contributor
Kevin started taking PCs apart in the 90s when Quake was on the way and his PC lacked the required components. Since then…
I have Meta Quest 3S and this is the best VR accessory yet — it’s on sale
Kiwi Design best VR accessory Meta Quest 3 headstrap

Ahead of the holidays and some prime family time, I've picked up the Meta Quest 3S. So far, my family is absolutely loving it, and my kids are constantly bugging me to play games, explore worlds, and get virtual. Naturally, I manage the time we're all spending -- you don't want too much screen time. But regardless, every one of us ends up becoming immersed, which means spending a lot of time with the headset on. One of the biggest drawbacks of the stock setup is that the headstrap is uncomfortable, and it puts a lot of pressure on your face. That means, the Meta Quest 3S's best VR accessory -- and the Meta Quest 3, too -- is a new, custom strap.

We grabbed the Kiwi Design Quest 3-Quest 3S headstrap and it's fantastic. It's also on sale right now for 20% off. Normally $30, it's discounted to $24 with a coupon code. Why am I sharing? If you pick up a Meta Quest 3 or 3S for yourself, or you're planning to gift one to someone over the holidays, I highly recommend ordering one of these straps. It vastly improves the experience and makes wearing the headset much more comfortable. It's also easy to adjust the fit, which is a big deal for kids. You have no idea how frustrating it was to constantly adjust the headstrap for my children between each turn.

Read more
Best early GPU Black Friday deals: Save on top graphics cards now
The Gigabyte RX 6750 GRE graphics card over a dark background.

Building a PC from scratch can be a lot of fun, and with the upcoming Black Friday on November 29, it's a perfect time for you to pick up hardware. One of the most fun bits of any build is picking the parts, and for that, graphics cards are probably the most fun to pick between. That said, GPUs also tend to be the most expensive pieces of hardware that go into a desktop, especially if you're trying to aim for something in the mid-to-high-end range that can easily reach $500 or even $1,000. That's why we've gone out and collected some of our favorite early Black Friday GPU deals for you below.
GIGABYTE NVIDIA GeForce RTX 3060 -- $290 $350 17% off

This RTX 3060 is a great starter card for those who want to be on a budget and will handle most slightly older games pretty well at 1080p and 60Hz, potentially up to 100. It may struggle a bit with newer titles without compromises, but that's fine given the reduced $290 price point.

Read more
Nvidia just scaled down DLSS 3, and that’s a good thing
The RTX 4080 Super graphics card sitting on a pink background.

Nvidia's signature tech, DLSS 3, just got yet another update -- and although it's subtle, it actually seems like a good thing for some of the best graphics cards. The latest version, 3.8.10, bundled with the GeForce 566.14 driver, doesn't seem to introduce any major changes, but Nvidia enthusiasts noticed that it's about half the size that it used to be. Where's that difference coming from?

No, Nvidia didn't downgrade DLSS 3 -- at least not in any major way. Although this hasn't been confirmed by Nvidia itself, it appears that the company removed a whole bunch of DLSS presets and replaced them with just two. These presets make it easier for gamers to choose the type of focus they want to apply to each game.

Read more