Skip to main content

MIT researchers develop simulation to manipulate real-world objects in videos

Pokémon Go just passed the 100 million player mark — and it’s still growing. The game’s success can be largely credited to millennial nostalgia but also to the app’s clever use of augmented reality (AR), which overlays the Pokémon universe on the real world via the smartphone’s camera.

But, even though you can toss a virtual Poké Ball at a virtual Pikachu on the sidewalk, traditional augmented reality doesn’t let virtual objects interact with objects in the real world. Pikachu can’t electrify a puddle, bump into a signpost, or jump up on the curb.

Recommended Videos

A team of Massachusetts Institute of Technology researchers led by Abe Davis, a Ph.D student at MIT’s Computer Science and Artificial Intelligence Laboratory (CSAIL), and Justin Chen, a post-doctoral associate in its Department of Civil and Environmental Engineering, might help smooth these distinctions between augmented reality and the real world by developing Interactive Dynamic Video (IDV), which they think can change industries from engineering to entertainment.

Two years ago, Davis and his team recorded silent videos of objects that vibrated when they were hit by sound. Using a processing algorithm, the researchers were able to recover music and speech from these tiny vibrations — effectively eavesdropping through a bag of potato chips. This discovery enticed Davis to explore vibrations further and inspired him to see if an algorithm could re-create an object’s movement by analyzing the object’s supersmall vibrations.

“There are some classic techniques used in engineering and physically based animation to simulate objects, but they depend on a full 3D model,” Davis told Digital Trends. “I got this crazy idea that you could do something similar with a video instead, and it ended up working surprisingly well.”

Davis and his team used a traditional camera to record various objects as they vibrated ever so slightly. They recorded a bush as it blew in the breeze and a guitar string as it was lightly touched. The algorithm then identified “vibrations modes” — how a particular object might sway this way or rock that way when disrupted by vibrations. From here, the researchers can use a simulation to virtually interact with the object.

So how could this apply to Pokémon Go?

Pokemon GO and Interactive Dynamic Video

“There is a broad range of applications for this work,” Davis said, “from monitoring the safety of bridges and buildings, to product testing, to special effects for movies, and ‘dynamic augmented reality,'” which can interact with and respond to virtual forces.

Davis admits, however, that it will still be a while before this technology ends up on mobile devices. Instead, it will likely be applied to movie special effects, as well as structural health monitoring to predict the safety of old bridges.

So, although we won’t soon see Pikachu virtually jump-start our car, we may see IDV help make our world a little more safer.

Dyllan Furness
Former Contributor
Dyllan Furness is a freelance writer from Florida. He covers strange science and emerging tech for Digital Trends, focusing…
The robot takeover comes another step closer — at Amazon
An Amazon robot working inside one of the company's warehouses.

Amazon is close to having more robots operating inside its warehouses than humans after the e-commerce giant announced this week that it now has more than a million robots working at its facilities around the world.

Over the years, Amazon has spent billions of dollars on the development and deployment of warehouse-based robots, which handle an array of tasks once performed by human workers.

Read more
This Lenovo ThinkPad laptop is over $1,400 off — hurry while stocks last!
The Lenovo ThinkPad T14 Gen 5 Intel laptop on a white background.

Now's an excellent time to take advantage of laptop deals from Lenovo, which has slashed the prices of a wide range of devices for its Black Friday in July sale. Lenovo's ThinkPad laptops are up to 45% off, and here's one of the most interesting offers available with such a discount — the Lenovo ThinkPad T14 Gen 5 at $1,440 off its estimated value of $3,199, so you'll only have to pay $1,759. That's an excellent price for this fantastic productivity tool, but you're going to have to push forward with your purchase as soon as possible because stocks may run out at any moment.

BUY NOW

Read more
Early Prime Day deal: Samsung’s 27-inch Odyssey G3 at its annual low price
Samsung Odyssey G3 gaming monitor on desk with keyboard and headset.

If you're ready to upgrade your monitor, this Samsung deal over at Amazon just might be your best bet. The 27-inch version of Samsung's Odyssey G3 is $130 right now, a full $100 off its regular $230 price and its lowest price of the year. It's a part of early Prime Day deals and a good sampling of what we can expect for the shopping holiday, which officially lands on July 8th. Tap the button below to see it for yourself or keep reading to see why we like this deal and why this should be your next monitor.

Buy Now

Read more