To better train drones and reduce the risk of damage to itself and its surroundings, MIT engineers developed a training platform called “Flight Goggles” based on virtual reality. This enables a fast-flying drone to train within a virtual environment while speeding through empty physical space. Given the nature of VR, these drones can now safely train for any environment and condition.
Without Flight Goggles, drone training typically includes a large enclosed area with nets to catch “careening” vehicles and physical props including doors and windows. If they crash, that’s an added expense to the project due lost time, repairs, or a complete drone replacement. This type of training is ideal for slow-moving drones designed to scan an environment, not fast-moving models.
“The moment you want to do high-throughput computing and go fast, even the slightest changes you make to its environment will cause the drone to crash,” says Sertac Karaman, associate professor of aeronautics and astronautics at MIT. “You can’t learn in that environment. If you want to push boundaries on how fast you can go and compute, you need some sort of virtual-reality environment.”
To develop Flight Goggles, the team began with a “hangar-like gymnasium” lined with motion-capture cameras mounted on the walls to track the drone’s movement through physical space. This data is inserted into an image rendering program that generates a photorealistic virtual environment based on the drone’s position and perspective. The program then sends that combined data back to the drone.
According to Karaman, the drone’s camera isn’t on, and instead “hallucinates” as it “sees” one environment while speeding through another, processing that visual feed at 90 frames per second. The drone used to test Flight Goggles was based on a 3D-printed nylon and carbon fiber frame, a custom-built circuit board, an embedded “supercomputer,” an inertial measurement unit, and a camera.
For the initial test, the team created a virtual living room with a window twice the size of the drone. Flying at five miles per hour, the vehicle darted through the virtual window 361 times and “crashed” only three times. All throughout this test, the team tweaked its navigation algorithm so the drone could “learn on the fly” and avoid virtual walls.
Of course, had the team used props instead of VR in this experiment, three repairs or complete drone replacements would be in order. But with Flight Goggles, the drone could “crash” thousands of times and the training would continue without costly repairs and downtime.
But you can’t have a VR training session without testing the drone in a real-world scenario. The team built the same window within the facility, and then turned on the drone’s on-board camera. The result: It zipped through the physical window 119 times and crashed/required human intervention six times.
While that doesn’t sound entirely successful, remember that the fast-flying drone learned to fly in virtual space not to mention zooming through the opening at 5 miles per hour. Karaman believes Flight Goggles could even safely train drones to fly around humans.