Skip to main content

Like Braille for your ears, Action Audio wordlessly translates sports into sound

Action Audio

THWOK. Ponk. Thwok. Ponk. THWOK. PONK. Thwok. Cheering intensifies. Ponk. THWOK. The squeak of sneakers on asphalt. PONK. Pause. Enthusiastic applause.

Recommended Videos

Recognize it? Of course you don’t. It is, in fact, Roger Federer’s “impossible shot” against Andre Agassi in the quarterfinals of the 2005 Dubai Tennis Championships. It’s about as perfect a moment of tennis as one could hope to witness, rendered unintelligible by the fact that it’s reproduced here as an abstract soundscape.

As easy to follow as tennis is visually, on an audioscape level, it’s as tough to follow as an episode of Twin Peaks you’ve switched on halfway through. That’s a challenge engineers from Tennis Australia, digital design and communications agency AKQA, and Monash University in Melbourne, Australia are working to solve with something called Action Audio. It’s an online audio stream that was developed for last month’s Australian Open broadcasts to assist the 285 million people globally — many of whom are tennis fans — living with blindness or a visual impairment.

“An easy way to understand it is … are you familiar with audio descriptions?” Tim Devine, AKQA’s executive creative director, told Digital Trends. “A lot of blind and low-vision people use them. This is kind of an abstract version of audio descriptions to give people more information about what’s happening [on the court].”

So commentary, basically?

There is, of course, a kind of audio description that’s existed in sport for years — and it’s the same one that allows sports fans to listen to sport on the radio, where images aren’t exactly in plentiful supply. But audio commentary from pundits isn’t the same thing as being able to watch a game and come to your own conclusions. The idea of Action Audio is to take images of the actions on court, and then to turn these into an unobtrusive action cue language — with different sounds to illustrate forehands, backhands, nail-bitingly close shots, and more.

“[We wanted people to be able to] make their own appraisal of what’s happening on the court,” Devine said. “So they can have their own insights, rather than just having the insights created by a commentator. They have more information. They’re able to say, ‘wow, Federer’s really hammering the backhand tonight.’ Without Action Audio, they couldn’t have that insight because no one’s telling them ‘that’s a backhand’ or ‘that’s a forehand.’ But you can do that very easily with abstract audio signals.”

Action Audio - Sample

The technology works by using 3D spatial data from a high-speed camera/computer vision system called Hawk-Eye, which tracks ball position in real time for electronic line calling for scoring purposes. This information is then transformed into 3D audio, turning actions into individual sound cues, while also shifting where the sound appears to be coming from in order to allow fans to get a sense of where the ball is on the court at any particular moment. “We deconstruct sport to reconstruct it as sound,” Devine said.

He noted that the project was inspired by neuroscientist David Eagleman’s sensory substitution technology. Eagleman, who Digital Trends interviewed last year, has long explored sensory substitution that’s able to, for instance, capture sound and turn it into vibratory patterns on a wearable vest or wristband. This could be used to allow a deaf person to understand what a person is saying when they speak.

“If you think about it, every signal that comes into the body, no matter what sense it is, is abstract, right?” Devine said. “We just learn to decode that signal. So there’s no reason why we can’t create — [and learn to understand] — an abstract signal that comes through an audio channel.”

Beyond the Australian Open

For the Australian Open, Action Audio was available as a television audio channel as part of the broadcast. Audio cues, which could be listened to in isolation or with audio commentary, offered sounds for forehand and backhand, along with others to indicate how close the ball is to the line. Devine said that, in the future, there are opportunities to expand this — the same way that any spoken language can be used to communicate simple intentions, as well as far more detailed observations. In the future, it may even be possible for users to customize the system for themselves, either changing the specific audio cues or, potentially, adding or removing detail depending on what they want to see highlighted. The possibilities are endless.

Devine acknowledged that this is a challenging job, not just technologically, but also in ensuring they are improving the viewing experience, not detracting from it. “Hearing for a blind or low-vision person is a very meaningful and precious sense,” he noted. Elsewhere in the interview, he asserted that the team is “trying to reduce cognitive load” on users.

Having demonstrated the technology, they now hope to expand it to coverage of other tennis tournaments such as Wimbledon (which already uses some impressive A.I. tech), the French Open, and the U.S. Open. They also believe that they can expand to other sports as well — although this will raise new challenges.

“Tennis is actually a really good example of action audio, because it’s dynamic,” Devine said. “It has moments of stopping, and moments of full action where everyone is quiet. But every sport is quite different. How could we sonify surfing? How do we give people a sense of what a surfer is doing? What do we identify? How about multiplayer events like basketball? How do we know what the information that someone who’s blind or low vision would like to hear? That’s where a codesign process will work.”

There’s more work to be done, but these are certainly promising beginnings.

Topics
Luke Dormehl
Former Digital Trends Contributor
I'm a UK-based tech writer covering Cool Tech at Digital Trends. I've also written for Fast Company, Wired, the Guardian…
Mod Bikes launches a BMW-inspired, limited-edition sidecar e-bike
MOD Easy Sidecar Sahara limited edition e-bike.

Mod Bikes has introduced the Mod Easy Sidecar Sahara, a limited-edition sidecar-style cargo e-bike modeled on the 1940s BMW R75 Sahara motorcycle. On November 9, 2024, Mod will exhibit the Sidecar Sahara at the Electrify Expo at the Circuit of the Americas racetrack in Austin, Texas.
Mod does sidecars

The BMW R 75 Sahara, which inspired the limited-edition e-bike, is a highly collectible motorcycle. Mod Bikes is one of the few e-bike manufacturers that sells e-bikes with sidecars in the U.S., along with other styles. The versatile Easy Sidecar models transform in just a few minutes from a sidecar e-bike to a standard, two-wheel e-bike. However, the sidecar configuration is fun for carrying pets, children, small adults, and groceries or other cargo.

Read more
I’m convinced Pins and Aces golf outfits are great for everyone, not just athletes
Pins and Aces banner with clothing styles

Some clothing designs stand the test of time, others transcend fashion categories or groups, just like Pins & Aces. Allow me to explain. Primarily meant for golfers and athletes, Pins & Aces carries a wide variety of golf-themed gear from golf bags to unique club heads. I don't really play golf so that doesn't excite me much. Maybe it piques your interest? Anyway, what does interest me is the large collection of unique and snazzy apparel. They don't just carry polos, they also carry streetwear, hats, shorts, joggers, and even women's and youth apparel. Thanks to Black Friday and the holiday shopping season, a ton of Pins & Aces apparel is on sale right now too. Go check it out.
Shop Now
 
What to shop in this Pins & Aces holiday sale?

Your best bet is going to be browsing the sale items yourself. I will call out a few highlights and share good deals here. However, there's so much that I cannot cover it all. Especially if you're shopping for someone else, do yourself a favor and go peek.

Read more
This Costway outdoor pizza oven isn’t messing around — $169 off for Black Friday
Costway outdoor pizza oven on patio

Forget those indoor-friendly pizza ovens. Forget the low-flame and low-smoke options. Me, personally, I want the real deal. That's why when I came across the Costway outdoor pizza oven I knew I had to share. It comes with a pizza stone, for authentic stone-cooked pizza duh, and a waterproof cover to protect the grill. But here's the best part: It can fire-cook a 12-inch homemade pizza pie. Oh, yes. Normally $309, it's on sale today for $140, which saves you $169. That is an incredible deal, let alone for something like this that can handle the fire and brimstone. It's live already alongside some other great early Black Friday deals.

 
Why should I buy this Costway outdoor pizza oven for Black Friday?
The best thing to do when it's time to cook, in my humble opinion, is to head outside and light up my trusty Blackstone. It works great, but if I want to wood fire a pizza I'll need something else. That's precisely where this Costway outdoor pizza oven comes in handy. You can adjust the level of fire and smoke to create a perfectly cooked pizza for your family, charred or not. The dual thermometer makes it easy to adjust the temperature when it's lit. Moreover, the under storage allows you to place some wood, utensils, or whatever else you might need -- so you're not always traipsing back to your shed, house or garage.

Read more