Skip to main content

How imperceptible vibrations could take augmented reality to a new level

Pokemon GO and Interactive Dynamic Video
Pokémon Go is the biggest breakout hit of the year, and though it may be starting to slip from its colossal popularity peak, it’s still a very well played game that millions enjoy on a daily basis. Part of what made it so eye-catching is its augmented reality feature. But as cool as that is, having a psuedo-hovering Pokémon superimposed over the real world doesn’t feel very much like reality.

To make the game more immersive, we’d need some way for the pocket monsters to interact with the environment they’re in, and have it react back. How would that be possible? A research team at MIT believes it’s found a way — through the use of micro-vibrations.

“Essentially, we’re looking at different frequencies of vibration, which represent a different way that an object can move. By identifying those shapes and frequencies, we can predict how an object will react in new situations,” Abe Davis, the lead researcher on the project, told Digital Trends. Along with fellow researchers Justin Chen and Fredo Durand, they’ve built upon previous research they conducted on the concept of visual microphones, to draw even more data from standard video.

“This might be something that’s more suitable for Pokémon Go 4 or 5, than Pokémon Go 2.”

“One way to think about it, is if I point my camera at a bush and I watch the wind rustle that bush for a whole minute, I’m watching a bunch of tiny movements of the bush, which are responses to various forces,” Davis explained.

Those movements are categorized as vibrations operating at various frequencies. Then, software can take the video and analyze those vibrations. It can figure out the types of forces at play to create those movements, and then guess at how larger forces, or different combinations of those same forces, may make the object react.

By recording the bush’s reaction to the wind, the software can eventually figure out how it might react to a brick — or Pikachu.

Bringing pocket monsters to life

Extrapolating more than just visual data from video became a focus of Davis’ interest throughout his time at MIT, and it was ultimately the core of his dissertation. However, explaining just how visual data from a video can be used beyond the norm isn’t easy. When Pokémon Go was released, he saw a great way to break it down.

Davis is a Pokémon Go player, having reached level 19 at the time we conducted our interview. We were even introduced to his most powerful Pokémon — Fluffles, a CP 1,592 Arcanine, who’s been tearing up the gyms in his local area. Fluffles was caught at the SIGGRAPH conference where Davis and his fellow researchers first showed off vibration model technology.

To use Pokémon as a showcase, Davis set up his phone on a tripod pointing in a specific direction. He then proceeded to catch a Pokémon, and then captured footage from that exact same position.

“I caught it and recorded about a minute of video, using the tripod for stability. I took that video back and processed it using the code I had written,” and the result was the video you see above.

A bush from the real world, which reacts (somewhat realistically) with a digital creation, is much closer to the sort of augmented reality future we’ve all been promised. Indeed, it even goes further than some of the things we’ve seen with Microsoft’s Hololens and the Magic Leap. Because of that, we shouldn’t expect this sort of technology to appear in the next Pokémon Go patch, or even its sequel.

“This might be something that’s more suitable for Pokémon Go 4 or 5, than Pokémon Go 2,” Davis cautioned.

That said, Davis and his fellow researchers had been working on this well before Pokémon Go was released, and there are many other potential applications for this technology beyond catching pocket monsters.

Shaking seconds off rendering CGI

What if, instead of rendering are entire explosion of an object or building, film makers could simply record video of an object and utilize this sort of algorithm to create a barebones animation? This has the potential to save huge chunks of time.

Of course, the artificially created movement that Davis has shown doesn’t look quite as good as the latest CGI blockbuster, but that’s not due to the weakness of the technique. Davis simply isn’t an artist. He has no idea how to polish his algorithm’s results.

“If you gave this tool to the world’s best artists, I suspect you could make it look really good.”

“The most expensive CGI is the most expensive CGI, because you pay the most expensive artists to do the most expensive art,” Davis said, jokingly. “If you gave this tool to the world’s best artists, I suspect you could make it look really good.”

“It’s about giving artists the best starting point. That’s how a lot of technology and special effects are used. If you want to make something look really good, you don’t want a canned solution. You want your artists to dictate every aspect of the look and feel of the final product.”

Another exciting use for the technology may be found in architecture, as well as insurance, where the tech could be used for structural health monitoring. Vibration modes and frequencies are already used in that profession, but they utilize much more complicated capture techniques to acquire the data.

“Typically that data is captured through lasers and accelerometers that have to placed on the object. The big advantage [with my technique], is that it’s very easy to point a camera at a building, but it’s pretty hard to paint a whole building with accelerometers or laser points,” said Davis. “This offers a convenient way to capture slightly lower quality data, which is great to figure out where you need to focus your attention.”

vibrationsbuilding
Abe Davis

If a company can test a building’s structural integrity by just recording some video of it and throwing an algorithm at it, it’d be possible for an intern with a camera to do work that’d previously demand a team of engineers.

Frame rates, resolution and magnification

Obviously, a commercial camera is much cheaper, easier to acquire and easier to operate than the technology this technique could help supplant. But there are certain hardware requirements that have a big effect on how well the algorithm works.

As with most video, a tripod is essential. While it wouldn’t be too difficult to separate out vibrations that effect the entire video, versus those that effect subjects within it, that’s a step that can be practically eliminated by using a sturdy vantage point for the camera to rest.

The type of camera, and its quality, can be important, too.

“The frame rate of the camera can actually determine what frequencies you can recover,” Davis said. “If you’re doing special effects, the frequencies you want to simulate are the frequencies that you can see, so frame rate isn’t so important. However, if you wanted to simulate a detailed solid object, then having higher frequencies which are captured at a higher frame rate is going to help.”

In one instance, Davis and his team wanted to track the vibrations from a Ukelele. But because of the way the strings on such an instrument vibrate, it was very important to use a high-frame-rate camera.

Conclusion

With all of the potential uses of the video vibration analysis work that Davis and his peers have been conducting, where does the technology go from here?

Although Davis plans to continue working on it in the future, he doesn’t have any immediate plans to leverage it for financial gain. There will be no micro-vibrations-from-video start up that Google or some other mega-corporation buys out in the near future. Part of that is because MIT owns the patent, having defensively applied for it.

However, you have to imagine that the likes of Microsoft and Magic Leap will be keeping an eye on this sort of technology, as it could be great for augmented reality.

Davis himself has now finished his dissertation, a comprehensive paper on all of his MIT conducted research, and will be graduating this September, before moving on to Stanford University for his post-doctorate.

For more information on any of Davis’ research, you can find all of his papers and studies on his official site. He also covered several aspects discussed here in his Ted Talk.

Editors' Recommendations

Jon Martindale
Jon Martindale is the Evergreen Coordinator for Computing, overseeing a team of writers addressing all the latest how to…
This Lenovo gaming laptop with an RTX 4050 is 31% off right now
The Lenovo Slim 5i facing forward.

One of the better gaming laptop deals for the holiday season comes from Lenovo. Today, you can buy the Lenovo Slim 5i Gen 8 gaming laptop for $931 meaning you save $419 off the regular price of $1,350. That's 31% off. Granted, Lenovo's estimated value prices tend to be a little higher than average, so the discount may be slightly smaller if real MSRP is taken into account. Even still, $931 for a gaming laptop with these specs is pretty special. Here’s what else you need to know about it before you buy.

Why you should buy the Lenovo Slim 5i
As one of the best laptop brands, Lenovo has a particular talent for making great gaming laptops. This model has a 13th-generation Intel Core i5 processor along with 16GB of memory and 512GB of SSD storage. It also has an Nvidia GeForce RTX 4050 graphics card which is well paired with its 16-inch WUXGA screen. The screen offers a resolution of 1920 x 1200, 45% NTSC, 300 nits of brightness, and a refresh rate of 144Hz so it’s perfect for this kind of spec.

Read more
This Samsung 32-inch 4K gaming monitor is 30% off for the holidays
The front view of the Samsung Odyssey Neo G7 4K curved gaming monitor.

Is your gaming setup still stuck with an old screen? If you've just upgraded with gaming PC deals, then you should maximize your machine's capabilities by investing in a gaming monitor like the 32-inch Samsung Odyssey Neo G7. Making this curved gaming monitor an even better buy is Samsung's $400 discount that brings its price down to $900 from $1,300. It's still not cheap, but it's the display that you need to fully appreciate the graphics of modern video games. You're going to have to hurry though, as stock may run out quickly for the holiday season.

Why you should buy the 32-inch Samsung Odyssey Neo G7 4K curved gaming monitor
The Samsung Odyssey Neo G7 4K curved gaming monitor delivers exceptional image quality, which is actually something that you'd expect from a screen that's made by one of the best TV brands. With 4K Ultra HD resolution, you'll enjoy lifelike details on the best PC games, and with a refresh rate of up to 165Hz, gameplay will be seamless with smooth movements. The 1000R curvature on the gaming monitor's 32-inch screen mimics the curve of the human eye so it fills your peripheral vision, and its support for AMD's FreeSync Premium Pro will further improve immersion by eliminating screen tearing and stuttering.

Read more
This Asus gaming laptop is discounted from $1,430 to $900
Asus ROG Zephyrus G14 2023 front view showing display and keyboard deck.

If you're on the hunt for gaming laptop deals, here's one that should catch your interest -- the Asus ROG Zephyrus G14 for just $900 from Best Buy, following a $530 discount on its sticker price of $1,430. It's still not as affordable as the cheapest laptop deals, but you should expect to pay a premium for a dependable gaming machine like this one. You'll be getting amazing value for money with the potential savings in this offer, but you need to hurry because there's no telling when the bargain will end.

Why you should buy the Asus ROG Zephyrus G14 gaming laptop
If you want a gaming laptop that will be able to run the best PC games at their highest settings, the Asus ROG Zephyrus G14 is a stellar option. It's powered by the AMD Ryzen 7 7735HS processor, the Nvidia GeForce RTX 4050 graphics card, and 16GB of RAM. The gaming laptop also comes with a 512GB SSD, which will provide ample storage space for multiple AAA titles and all of the updates that you need to download for them. The Asus ROG Zephyrus G14 also comes with Windows 11 Home out of the box, so you can start installing games right after booting it up for the first time.

Read more