Last summer, a team of researchers from MIT, Microsoft and Adobe discovered a way to reproduce audio by analyzing microscopic vibrations in recorded video footage. In the past few months, MIT Ph.D. candidate Abe Davis and his team took this video algorithm to another level: predicting how objects will move, according to Mashable.
Article updated by Chris Palermino 3/20/15: MIT technology has new implications
In a TED Talk on Monday in Vancouver, Davis demonstrated how the new algorithm works with a silent video of a bush moving gently in the breeze. After converting the file with this new software into a photo-video file, he clicked and prodded at a ‘still’ image of the bush with his cursor. His mouse created a realistic, virtual ‘wind,’ showing how the bush would move when it was blown.
While still in beta, this technology has a myriad of applications in fields like engineering, transportation and gaming. One example that has been suggested is using the algorithm to help pinpoint stress points in a bridge’s design. Imagine if this could help predict potential rail issues on trains, structural issues in buildings, or the ramifications of that last punch you threw in Tekken?
Original article from 8/5/14 by Drew Prindle
No audio? No problem. A team of researchers from MIT, Microsoft, and Adobe has developed an algorithm that allows them to reconstruct an audio signal even when only visual information is available. Using nothing more than a high speed camera and a special processing algorithm, the team was able to extract the audio signals in a room from 15 feet away through soundproof glass.
How? Well when you get down to it, sound waves are really just tiny disturbances in the air. Therefore, when sound waves strike something delicate –say, a piece of tin foil, a bag of chips, or the leaves of a house plant– they cause the object to vibrate ever so slightly. It’s just like how the rear-view mirror vibrates when your buddy turns cranks the subwoofer on his car stereo, just on a much more minute scale. The waves that occur when you’re just having a conversation are much weaker, and tend to cause more minute vibrations. To the naked eye, these disturbances are practically imperceptible — but with the help of high-speed photography, the team was able to capture movements as small as a tenth of a micrometer, and then use that information to guesstimate and rebuild the audio signal. Check out the video to see it in action:
As if that wasn’t incredible enough, the team also a demonstrated variation on the algorithm that allows them to extract sound from ordinary 60 frame per second video footage. Generally speaking, the sensors on most digital cameras are designed to scan images horizontally, one row at a time. Normally, that’s not a problem, but when you’re shooting fast-moving objects, this can sometimes leads to odd visual artifacts. The team was able to exploit this technological quirk to tease out information about the objects’ high-frequency vibration and, once again, use that info to reconstruct a usable (albeit murky) audio signal. It’s not quite as clear as the audio signal ripped from the high-speed camera, but even so, the fact that this kind of reconstruction is possible is mind blowing.
The researchers are presenting their work at the computer graphics conference Siggraph this month. Find out more here.
