MIT research may revolutionize structural engineering with a camera that ‘sees’ sound

mit audio from video algorithm listenmit
Last summer, a team of researchers from MIT, Microsoft and Adobe discovered a way to reproduce audio by analyzing microscopic vibrations in recorded video footage. In the past few months, MIT Ph.D. candidate Abe Davis and his team took this video algorithm to another level: predicting how objects will move, according to Mashable.

Article updated by Chris Palermino 3/20/15: MIT technology has new implications

In a TED Talk on Monday in Vancouver, Davis demonstrated how the new algorithm works with a silent video of a bush moving gently in the breeze. After converting the file with this new software into a photo-video file, he clicked and prodded at a ‘still’ image of the bush with his cursor. His mouse created a realistic, virtual ‘wind,’ showing how the bush would move when it was blown.

While still in beta, this technology has a myriad of applications in fields like engineering, transportation and gaming. One example that has been suggested is using the algorithm to help pinpoint stress points in a bridge’s design. Imagine if this could help predict potential rail issues on trains, structural issues in buildings, or the ramifications of that last punch you threw in Tekken?

Original article from 8/5/14 by Drew Prindle

No audio? No problem. A team of researchers from MIT, Microsoft, and Adobe has developed an algorithm that allows them to reconstruct an audio signal even when only visual information is available. Using nothing more than a high speed camera and a special processing algorithm, the team was able to extract the audio signals in a room from 15 feet away through soundproof glass.

How? Well when you get down to it, sound waves are really just tiny disturbances in the air. Therefore, when sound waves strike something delicate –say, a piece of tin foil, a bag of chips, or the leaves of a house plant– they cause the object to vibrate ever so slightly. It’s just like how the rear-view mirror vibrates when your buddy turns cranks the subwoofer on his car stereo, just on a much more minute scale. The waves that occur when you’re just having a conversation are much weaker, and tend to cause more minute vibrations. To the naked eye, these disturbances are practically imperceptible — but with the help of high-speed photography, the team was able to capture movements as small as a tenth of a micrometer, and then use that information to guesstimate and rebuild the audio signal. Check out the video to see it in action:

As if that wasn’t incredible enough, the team also a demonstrated variation on the algorithm that allows them to extract sound from ordinary 60 frame per second video footage. Generally speaking, the sensors on most digital cameras are designed to scan images horizontally, one row at a time. Normally, that’s not a problem, but when you’re shooting fast-moving objects, this can sometimes leads to odd visual artifacts. The team was able to exploit this technological quirk to tease out information about the objects’ high-frequency vibration and, once again, use that info to reconstruct a usable (albeit murky) audio signal. It’s not quite as clear as the audio signal ripped from the high-speed camera, but even so, the fact that this kind of reconstruction is possible is mind blowing.

The researchers are presenting their work at the computer graphics conference Siggraph this month. Find out more here.

Emerging Tech

With cameras that know dogs from Dodges, Honda is making intersections safer

Honda and the city of Marysville, Ohio are working on creating a smart intersection. The goal would not only help better direct the flow of traffic, it could also help save the lives of pedestrians and cyclists.
Photography

Adobe Premiere Rush CC is the cloud-based video editing app you've been waiting for

On stage at Adobe MAX 2018, Adobe announced its cloud-centric, social video-editing application, Adobe Premiere Rush CC. We took some time to put it through its paces to see what it offers, how it works, and what's missing.
Photography

Adobe’s Premiere Rush is a video-editing app designed for social media projects

At Adobe MAX 2018, Adobe unveiled updates across the board for all of its Creative Cloud apps, from the release of Premiere Rush CC, a social-focused video editor, to Project Gemini, a digital drawing and painting tool.
Movies & TV

Stream till you scream with the best scary movies on Netflix, Hulu, and more

Need some ghoulish entertainment to get in the Halloween spirit? Check out some of the best horror movies streaming now. Whether you like creepy atmosphere or bloody jump scares, you'll find something to spook you.
Movies & TV

The best movies on Netflix in October, from 'The Witch’ to ‘Black Panther’

Save yourself from hours wasted scrolling through Netflix's massive library by checking out our picks for the streamer's best movies available right now, whether you're into explosive action, subdued humor, or anything in between.
Emerging Tech

Boston Dynamics is trying to make fetch happen with its new working robot dog

Boston Dynamics wants to see Spot in the workplace, but not as part of take-your-dog-to-work days. Quite the opposite, in fact, as the technology company believes its extraordinary robo-dog is now ready to start work.
Emerging Tech

Regular paints and plastics will soon be able to ‘heal’ like skin

Imagine if paints, plastics, or other coatings could heal up like human skin in the event that they suffered damage. Thanks to researchers at Clemson University, such technology is almost here.
Emerging Tech

Here’s how Microsoft’s Hololens is helping NASA build the new Orion spacecraft

Lockheed Martin is turning to Microsoft’s mixed reality Hololens smartglasses to help build NASA's Orion spacecraft, which could one day help rocket astronauts as far afield as Mars.
Emerging Tech

Shrimp eyes inspire new camera focused on helping self-driving cars see better

By mimicking the vision of mantis shrimp, researchers were able to make significant improvements on today’s commercial cameras. They hope their technology can help mitigate accidents by letting self-driving vehicles see more clearly.
Emerging Tech

This intelligent parachute system can bail out clumsy drone pilots

Parachutes can save drones when they unexpectedly fall from the sky. Among a number of such systems, Austrian firm Drone Rescue is this week showing off its latest design that automatically deploys when it senses trouble.
Cars

‘Bloodhound’ rocket car needs a speedy cash injection to survive

The rocket-powered Bloodhound car has driven into difficulties, with the company behind the project needing a multi-million-dollar cash injection to save its dream of attempting a 1,000 mph land speed record.
Emerging Tech

Tokyo robotic warehouse needs almost no human workers

Uniqlo has unveiled its first robot-powered warehouse that requires 90 percent fewer human workers to operate. The Japanese clothing giant plans to invest close to $1 billion dollars to convert all of its warehouses worldwide.
Emerging Tech

Curious how A.I. 'brains' work? Here's a super-simple breakdown of deep learning

What is deep learning? A branch of machine learning, this field deals with the creation of neural networks that are modeled after the brain and adept at dealing with large amounts of human-oriented data, like writing and voice commands.
Emerging Tech

Drop everything and watch Boston Dynamics’ robo-dog dance to ‘Uptown Funk’

After a few years of Earthbound training, Boston Dynamics’ SpotMini robot dog is ready to take on Mars. Bruno Mars, to be precise. Check out Skynet's future pet as you've never seen it before.