Skip to main content

Like Braille for your ears, Action Audio wordlessly translates sports into sound

Action Audio

THWOK. Ponk. Thwok. Ponk. THWOK. PONK. Thwok. Cheering intensifies. Ponk. THWOK. The squeak of sneakers on asphalt. PONK. Pause. Enthusiastic applause.

Recognize it? Of course you don’t. It is, in fact, Roger Federer’s “impossible shot” against Andre Agassi in the quarterfinals of the 2005 Dubai Tennis Championships. It’s about as perfect a moment of tennis as one could hope to witness, rendered unintelligible by the fact that it’s reproduced here as an abstract soundscape.

As easy to follow as tennis is visually, on an audioscape level, it’s as tough to follow as an episode of Twin Peaks you’ve switched on halfway through. That’s a challenge engineers from Tennis Australia, digital design and communications agency AKQA, and Monash University in Melbourne, Australia are working to solve with something called Action Audio. It’s an online audio stream that was developed for last month’s Australian Open broadcasts to assist the 285 million people globally — many of whom are tennis fans — living with blindness or a visual impairment.

“An easy way to understand it is … are you familiar with audio descriptions?” Tim Devine, AKQA’s executive creative director, told Digital Trends. “A lot of blind and low-vision people use them. This is kind of an abstract version of audio descriptions to give people more information about what’s happening [on the court].”

So commentary, basically?

There is, of course, a kind of audio description that’s existed in sport for years — and it’s the same one that allows sports fans to listen to sport on the radio, where images aren’t exactly in plentiful supply. But audio commentary from pundits isn’t the same thing as being able to watch a game and come to your own conclusions. The idea of Action Audio is to take images of the actions on court, and then to turn these into an unobtrusive action cue language — with different sounds to illustrate forehands, backhands, nail-bitingly close shots, and more.

“[We wanted people to be able to] make their own appraisal of what’s happening on the court,” Devine said. “So they can have their own insights, rather than just having the insights created by a commentator. They have more information. They’re able to say, ‘wow, Federer’s really hammering the backhand tonight.’ Without Action Audio, they couldn’t have that insight because no one’s telling them ‘that’s a backhand’ or ‘that’s a forehand.’ But you can do that very easily with abstract audio signals.”

Action Audio - Sample

The technology works by using 3D spatial data from a high-speed camera/computer vision system called Hawk-Eye, which tracks ball position in real time for electronic line calling for scoring purposes. This information is then transformed into 3D audio, turning actions into individual sound cues, while also shifting where the sound appears to be coming from in order to allow fans to get a sense of where the ball is on the court at any particular moment. “We deconstruct sport to reconstruct it as sound,” Devine said.

He noted that the project was inspired by neuroscientist David Eagleman’s sensory substitution technology. Eagleman, who Digital Trends interviewed last year, has long explored sensory substitution that’s able to, for instance, capture sound and turn it into vibratory patterns on a wearable vest or wristband. This could be used to allow a deaf person to understand what a person is saying when they speak.

“If you think about it, every signal that comes into the body, no matter what sense it is, is abstract, right?” Devine said. “We just learn to decode that signal. So there’s no reason why we can’t create — [and learn to understand] — an abstract signal that comes through an audio channel.”

Beyond the Australian Open

For the Australian Open, Action Audio was available as a television audio channel as part of the broadcast. Audio cues, which could be listened to in isolation or with audio commentary, offered sounds for forehand and backhand, along with others to indicate how close the ball is to the line. Devine said that, in the future, there are opportunities to expand this — the same way that any spoken language can be used to communicate simple intentions, as well as far more detailed observations. In the future, it may even be possible for users to customize the system for themselves, either changing the specific audio cues or, potentially, adding or removing detail depending on what they want to see highlighted. The possibilities are endless.

Devine acknowledged that this is a challenging job, not just technologically, but also in ensuring they are improving the viewing experience, not detracting from it. “Hearing for a blind or low-vision person is a very meaningful and precious sense,” he noted. Elsewhere in the interview, he asserted that the team is “trying to reduce cognitive load” on users.

Having demonstrated the technology, they now hope to expand it to coverage of other tennis tournaments such as Wimbledon (which already uses some impressive A.I. tech), the French Open, and the U.S. Open. They also believe that they can expand to other sports as well — although this will raise new challenges.

“Tennis is actually a really good example of action audio, because it’s dynamic,” Devine said. “It has moments of stopping, and moments of full action where everyone is quiet. But every sport is quite different. How could we sonify surfing? How do we give people a sense of what a surfer is doing? What do we identify? How about multiplayer events like basketball? How do we know what the information that someone who’s blind or low vision would like to hear? That’s where a codesign process will work.”

There’s more work to be done, but these are certainly promising beginnings.

Topics
Luke Dormehl
I'm a UK-based tech writer covering Cool Tech at Digital Trends. I've also written for Fast Company, Wired, the Guardian…
Best grill deals: Summer savings on gas, charcoal, and pellet grills
People grilling outside.

Warm weather is here, and so is grilling season. Now is as good a time as any to set out in search of one of the best outdoor grills, and it’s never a bad time to keep an eye out for a discount on one as well. Whether you’re a grilling beginner ready for some burgers with the family or a seasoned pro looking to host summer gatherings around one of the best smart grills and smokers, we’ve tracked down all of the details you need to land a deal on a grill that’s right for your needs. Read onward for more details.
Everdure CUBE charcoal grill — $150, was $199

The Everdure CUBE charcoal grill is a unique little grill that can truly go almost anywhere with you. It’s lightweight and only 17-inches long, yet still can handle almost any grilling challenge a griller on the go could hope to throw at it. It has a built-in heat protection shield and chrome handles that remain cool while cooking, and the removable charcoal tray offers easy cleanup when you’re finished at the campground or on the beach. Wherever you intend to do some grilling this summer, the Everdure CUBE charcoal grill is a great option as a copilot.

Read more
Speed up your daily commute with these e-bike deals
ENGWE EP-2 Pro e-bike right side shot next to an unmowed field.

The best electric bikes have come a long way in recent years and we don't just mean in terms of the miles you travel. With technology changing quickly, there are also an increasing number of electric bike deals around and we've picked out some of the very best. Electric bikes are a great way of getting around more easily and without necessarily needing to use as much stamina as a regular bike. Below, you'll find fantastic e-bike bike deals from many different retailers. There'll be something for every need here. Let's take a look at them.
Jasion EB5 Electric Bike -- $430, was $750

With a 350w brushless motor, the Jasion EB5 Electric Bike can achieve a top speed of 20 MPH with a range of up to 25-40 miles per charge. It has four working modes with a PAS mode where the motor assists with a moderate amount of power as you pedal, along with a Pure Electric Mode, a Pedal Mode, and a Booster Mode where the bike assist on hills and anywhere else you might be struggling. 26-inch puncture-resistant tires provide excellent durability while there's a high-strength front fork suspension, Shimano 7-speed gears, dual lights, and an LCD display. The latter offers controls for adjustable backlight brightness with one of five levels, along with mode controls.

Read more
Jackery graduates from pioneer to sustainable master with latest launch
Jackery 2000 Plus used outdoors while camping

This content was produced in partnership with Jackery.
In 2012, at a critical time when sustainability efforts were ramping up everywhere, Jackery was born. The prevailing vision behind its creation was to provide accessible green energy to everyone. Years later, in 2016, Jackery launched the world's first outdoor-friendly portable power stations, and soon after, introduced the world's first portable solar panels. They would change many lives for the better but also transform an entire market that's now dedicated to green, sustainable energy solutions. Today, over 11 years later, Jackery is still going strong, and getting ready to launch one of its most advanced products to date, the Jackery Solar Generator 2000 Plus.

Ideal for outdoor applications, portability, and emergency backup usage, the Jackery Solar Generator 2000 Plus delivers some incredible features that honor Jackery's original mission -- "to make green energy accessible anytime and anywhere". More importantly, it offers safe and reliable power independence for all, thanks to a combined software and hardware defense system, which we'll explore in more detail below, and industry-leading fast-charge technology to get you working, playing, or back in the game as quickly as possible.
Learn More
 
Advanced technology for a better anywhere experience

Read more