Skip to main content

MIT research may revolutionize structural engineering with a camera that ‘sees’ sound

mit audio from video algorithm listenmit
Image used with permission by copyright holder
Last summer, a team of researchers from MIT, Microsoft and Adobe discovered a way to reproduce audio by analyzing microscopic vibrations in recorded video footage. In the past few months, MIT Ph.D. candidate Abe Davis and his team took this video algorithm to another level: predicting how objects will move, according to Mashable.

Article updated by Chris Palermino 3/20/15: MIT technology has new implications

In a TED Talk on Monday in Vancouver, Davis demonstrated how the new algorithm works with a silent video of a bush moving gently in the breeze. After converting the file with this new software into a photo-video file, he clicked and prodded at a ‘still’ image of the bush with his cursor. His mouse created a realistic, virtual ‘wind,’ showing how the bush would move when it was blown.

While still in beta, this technology has a myriad of applications in fields like engineering, transportation and gaming. One example that has been suggested is using the algorithm to help pinpoint stress points in a bridge’s design. Imagine if this could help predict potential rail issues on trains, structural issues in buildings, or the ramifications of that last punch you threw in Tekken?

Original article from 8/5/14 by Drew Prindle

No audio? No problem. A team of researchers from MIT, Microsoft, and Adobe has developed an algorithm that allows them to reconstruct an audio signal even when only visual information is available. Using nothing more than a high speed camera and a special processing algorithm, the team was able to extract the audio signals in a room from 15 feet away through soundproof glass.

The Visual Microphone: Passive Recovery of Sound from Video

How? Well when you get down to it, sound waves are really just tiny disturbances in the air. Therefore, when sound waves strike something delicate –say, a piece of tin foil, a bag of chips, or the leaves of a house plant– they cause the object to vibrate ever so slightly. It’s just like how the rear-view mirror vibrates when your buddy turns cranks the subwoofer on his car stereo, just on a much more minute scale. The waves that occur when you’re just having a conversation are much weaker, and tend to cause more minute vibrations. To the naked eye, these disturbances are practically imperceptible — but with the help of high-speed photography, the team was able to capture movements as small as a tenth of a micrometer, and then use that information to guesstimate and rebuild the audio signal. Check out the video to see it in action:

As if that wasn’t incredible enough, the team also a demonstrated variation on the algorithm that allows them to extract sound from ordinary 60 frame per second video footage. Generally speaking, the sensors on most digital cameras are designed to scan images horizontally, one row at a time. Normally, that’s not a problem, but when you’re shooting fast-moving objects, this can sometimes leads to odd visual artifacts. The team was able to exploit this technological quirk to tease out information about the objects’ high-frequency vibration and, once again, use that info to reconstruct a usable (albeit murky) audio signal. It’s not quite as clear as the audio signal ripped from the high-speed camera, but even so, the fact that this kind of reconstruction is possible is mind blowing.

The researchers are presenting their work at the computer graphics conference Siggraph this month. Find out more here.

Drew Prindle
Former Digital Trends Contributor
Drew Prindle is an award-winning writer, editor, and storyteller who currently serves as Senior Features Editor for Digital…
4 simple pieces of tech that helped me run my first marathon
Garmin Forerunner 955 Solar displaying pace information.

The fitness world is littered with opportunities to buy tech aimed at enhancing your physical performance. No matter your sport of choice or personal goals, there's a deep rabbit hole you can go down. It'll cost plenty of money, but the gains can be marginal -- and can honestly just be a distraction from what you should actually be focused on. Running is certainly susceptible to this.

A few months ago, I ran my first-ever marathon. It was an incredible accomplishment I had no idea I'd ever be able to reach, and it's now going to be the first of many I run in my lifetime. And despite my deep-rooted history in tech, and the endless opportunities for being baited into gearing myself up with every last product to help me get through the marathon, I went with a rather simple approach.

Read more
This bracelet helps you fall asleep faster and sleep longer

This content was produced in partnership with Apollo Neuroscience.
Have you been struggling to get the recommended seven hours of sleep? It's always frustrating when you get in bed at a reasonable time, then toss and turn for a hours before you actually sleep. The quality of that sleep is important too. If you're waking up multiple times during the night, you're likely not getting the quality REM cycle sleep that truly rejuvenates your body. If traditional remedies like herbal teas and noise machines just aren't helping, maybe it's time to try a modern solution. Enter the Apollo wearable.

Now we understand being a little skeptical. How can a bracelet on your wrist or ankle affect your sleep patterns? Certainly the answer to a better night's sleep can't be so simple. We considered these same things when we first heard of it. We'll dive deeper into the science behind the Apollo wearable, but suffice it to say that many people have experienced deeper, uninterrupted sleep while wearing one.
A non-conventional approach to better sleep

Read more
The 11 best Father’s Day deals that you can get for Sunday
Data from a workout showing on the screen of the Apple Watch Series 8.

Father's Day is fast approaching and there's still time to buy your beloved Dad a sweet new device to show him how much you love him. That's why we've rounded up the ten best Father's Day tech deals going on right now. There's something for most budgets here, including if you're able to spend a lot on your loved one. Read on while we take you through the highlights and remember to order fast so you don't miss out on the big day.
Samsung Galaxy Tab A8 -- $200, was $230

While it's the Plus version of the Samsung Galaxy Tab A8 that features in our look at the best tablets, the standard variety is still worth checking out. Saving your Dad the need to dig out their laptop or squint at a small phone screen, the Samsung Galaxy Tab A8 offers a large 10.5-inch LCD display and all the useful features you would expect. 128GB of storage means plenty of room for all your Dad's favorite apps as well as games too. A long-lasting battery and fast charging save him the need for a power source too often too.

Read more