Skip to main content

MIT research may revolutionize structural engineering with a camera that ‘sees’ sound

mit audio from video algorithm listenmit
Image used with permission by copyright holder
Last summer, a team of researchers from MIT, Microsoft and Adobe discovered a way to reproduce audio by analyzing microscopic vibrations in recorded video footage. In the past few months, MIT Ph.D. candidate Abe Davis and his team took this video algorithm to another level: predicting how objects will move, according to Mashable.

Article updated by Chris Palermino 3/20/15: MIT technology has new implications

In a TED Talk on Monday in Vancouver, Davis demonstrated how the new algorithm works with a silent video of a bush moving gently in the breeze. After converting the file with this new software into a photo-video file, he clicked and prodded at a ‘still’ image of the bush with his cursor. His mouse created a realistic, virtual ‘wind,’ showing how the bush would move when it was blown.

While still in beta, this technology has a myriad of applications in fields like engineering, transportation and gaming. One example that has been suggested is using the algorithm to help pinpoint stress points in a bridge’s design. Imagine if this could help predict potential rail issues on trains, structural issues in buildings, or the ramifications of that last punch you threw in Tekken?

Original article from 8/5/14 by Drew Prindle

No audio? No problem. A team of researchers from MIT, Microsoft, and Adobe has developed an algorithm that allows them to reconstruct an audio signal even when only visual information is available. Using nothing more than a high speed camera and a special processing algorithm, the team was able to extract the audio signals in a room from 15 feet away through soundproof glass.

The Visual Microphone: Passive Recovery of Sound from Video

How? Well when you get down to it, sound waves are really just tiny disturbances in the air. Therefore, when sound waves strike something delicate –say, a piece of tin foil, a bag of chips, or the leaves of a house plant– they cause the object to vibrate ever so slightly. It’s just like how the rear-view mirror vibrates when your buddy turns cranks the subwoofer on his car stereo, just on a much more minute scale. The waves that occur when you’re just having a conversation are much weaker, and tend to cause more minute vibrations. To the naked eye, these disturbances are practically imperceptible — but with the help of high-speed photography, the team was able to capture movements as small as a tenth of a micrometer, and then use that information to guesstimate and rebuild the audio signal. Check out the video to see it in action:

As if that wasn’t incredible enough, the team also a demonstrated variation on the algorithm that allows them to extract sound from ordinary 60 frame per second video footage. Generally speaking, the sensors on most digital cameras are designed to scan images horizontally, one row at a time. Normally, that’s not a problem, but when you’re shooting fast-moving objects, this can sometimes leads to odd visual artifacts. The team was able to exploit this technological quirk to tease out information about the objects’ high-frequency vibration and, once again, use that info to reconstruct a usable (albeit murky) audio signal. It’s not quite as clear as the audio signal ripped from the high-speed camera, but even so, the fact that this kind of reconstruction is possible is mind blowing.

The researchers are presenting their work at the computer graphics conference Siggraph this month. Find out more here.

Editors' Recommendations

Drew Prindle
Former Digital Trends Contributor
Drew Prindle is an award-winning writer, editor, and storyteller who currently serves as Senior Features Editor for Digital…
Don’t buy the Meta Quest Pro for gaming. It’s a metaverse headset first
Meta Quest Pro enables 3D modeling in mixed reality.

Last week’s Meta Connect started off promising on the gaming front. Viewers got release dates for Iron Man VR, an upcoming Quest game that was previously a PS VR exclusive, as well as Among Us VR. Meta, which owns Facebook, also announced that it was acquiring three major VR game studios -- Armature Studio, Camouflaj Team, and Twisted Pixel -- although we don’t know what they’re working on just yet.

Unfortunately, that’s where the Meta Connect's gaming section mostly ended. Besides tiny glimpses and a look into fitness, video games were not the show's focus. Instead, CEO Mark Zuckerberg wanted to focus on what seemed to be his company’s real vision of VR's future, which involves a lot of legs and a lot of work with the Quest Pro, a mixed reality headset that'll cost a whopping $1,500.

Read more
Meet the game-changing pitching robot that can perfectly mimic any human throw
baseball hitter swings and misses

Who’s your favorite baseball pitcher? Shane McClanahan? Sandy Alcantara? Justin Verlander? Whoever you said, two of the top sports-tech companies in the U.S. -- Rapsodo and Trajekt Sports -- have teamed up to build a robot version of them, and the results are reportedly uncannily accurate.

Okay, so we’re not talking about walking-talking-pitching standalone robots, as great a sci-fi-tinged MLB ad as that would be. However, Rapsodo and Trajekt have combined their considerable powers to throw a slew of different technologies at the problem of building a machine that's able to accurately simulate the pitching style of whichever player you want to practice batting against -- and they may just have pulled it off, too.

Read more
The best portable power stations
EcoFlow DELTA 2 on table at campsite for quick charging.

Affordable and efficient portable power is a necessity these days, keeping our electronic devices operational while on the go. But there are literally dozens of options to choose from, making it abundantly difficult to decide which mobile charging solution is best for you. We've sorted through countless portable power options and came up with six of the best portable power stations to keep your smartphones, tablets, laptops, and other gadgets functioning while living off the grid.
The best overall: Jackery Explorer 1000

Jackery has been a mainstay in the portable power market for several years, and today, the company continues to set the standard. With three AC outlets, two USB-A, and two USB-C plugs, you'll have plenty of options for keeping your gadgets charged.

Read more