Skip to main content

MIT researchers develop simulation to manipulate real-world objects in videos

Interactive Dynamic Video
Pokémon Go just passed the 100 million player mark — and it’s still growing. The game’s success can be largely credited to millennial nostalgia but also to the app’s clever use of augmented reality (AR), which overlays the Pokémon universe on the real world via the smartphone’s camera.

But, even though you can toss a virtual Poké Ball at a virtual Pikachu on the sidewalk, traditional augmented reality doesn’t let virtual objects interact with objects in the real world. Pikachu can’t electrify a puddle, bump into a signpost, or jump up on the curb.

A team of Massachusetts Institute of Technology researchers led by Abe Davis, a Ph.D student at MIT’s Computer Science and Artificial Intelligence Laboratory (CSAIL), and Justin Chen, a post-doctoral associate in its Department of Civil and Environmental Engineering, might help smooth these distinctions between augmented reality and the real world by developing Interactive Dynamic Video (IDV), which they think can change industries from engineering to entertainment.

Two years ago, Davis and his team recorded silent videos of objects that vibrated when they were hit by sound. Using a processing algorithm, the researchers were able to recover music and speech from these tiny vibrations — effectively eavesdropping through a bag of potato chips. This discovery enticed Davis to explore vibrations further and inspired him to see if an algorithm could re-create an object’s movement by analyzing the object’s supersmall vibrations.

“There are some classic techniques used in engineering and physically based animation to simulate objects, but they depend on a full 3D model,” Davis told Digital Trends. “I got this crazy idea that you could do something similar with a video instead, and it ended up working surprisingly well.”

Davis and his team used a traditional camera to record various objects as they vibrated ever so slightly. They recorded a bush as it blew in the breeze and a guitar string as it was lightly touched. The algorithm then identified “vibrations modes” — how a particular object might sway this way or rock that way when disrupted by vibrations. From here, the researchers can use a simulation to virtually interact with the object.

So how could this apply to Pokémon Go?

Pokemon GO and Interactive Dynamic Video

“There is a broad range of applications for this work,” Davis said, “from monitoring the safety of bridges and buildings, to product testing, to special effects for movies, and ‘dynamic augmented reality,'” which can interact with and respond to virtual forces.

Davis admits, however, that it will still be a while before this technology ends up on mobile devices. Instead, it will likely be applied to movie special effects, as well as structural health monitoring to predict the safety of old bridges.

So, although we won’t soon see Pikachu virtually jump-start our car, we may see IDV help make our world a little more safer.

Editors' Recommendations

Dyllan Furness
Dyllan Furness is a freelance writer from Florida. He covers strange science and emerging tech for Digital Trends, focusing…
Best Apple Studio Display deals: Up to 16% off the 5K monitor
Apple Mac Studio and Studio Display.

Graphic designers and creative professionals who are in the market for monitor deals should consider going for the Apple Studio Display, especially since you can now enjoy discounts on the premium screen. We've rounded up the best Apple Studio Display deals that are currently available right here so you can get the monitor at up to 16% off, but you'll have to proceed with your purchase as soon as possible because we're not sure when these offers will expire.
Today's best Apple Studio Display deals

Apple Studio Display (Standard Glass) --

Read more
How to recall an email in Gmail if you accidentally sent it
A person sitting at a desk, in front of a computer monitor with their head in their hands.

The instantaneous delivery of email comes with consequences. Once you send an email, it’s gone and out of your hands. We all make mistakes, though, and Google gets it. To help out, Gmail includes a feature called Undo Send that allows you to cancel your send request. In the past, you had to manually enable it, but now it’s on by default.

Here’s how to make the most of it.

Read more
Does Dell or HP make the best 16-inch laptop? You might be surprised
Photo of an HP Envy 16 laptop.

You can spend a lot of money on 16-inch laptops with fast components for video editing and photo editing. We're talking $2,500 and a lot more. But there's a class of midrange 16-inch machines that offer much of the same performance for less money.

Dell's Inspiron 16 Plus and HP's Envy 16 are two such laptops with prices that start well under $2,000. Both can be equipped with some fast components, but they're not identical by any means. Which of these more affordable 16-inch powerhouses is the best?
Specs and configurations

Read more