Skip to main content

New Zealand attack shows that as A.I. filters get smarter, so do violators

Facebook, YouTube, and Twitter are quick to share stats on how artificial intelligence filters are improving, but the aftermath of last week’s shooting in Christchurch, New Zealand made gaps in the system terrifyingly obvious. Videos shot from the shooter’s point of view were uploaded to social media in numbers at least in the hundred thousand range after a copy of the original was posted to a file-sharing platform called 8chan.

The attack on two mosques in Christchurch left 50 dead and another 50 wounded, according to the authorities. The 28-year-old shooter wore a helmet-mounted camera and livestreamed the shootings in a way that some are saying was “designed for maximum spread on social media.” YouTube’s chief product officer, Neal Mohan, says the shooting was uploaded faster, with more videos, than previous incidents.

Related Videos

Three days later, and social media platforms were still struggling to keep copies of the 17-minute video off the networks. YouTube temporarily erred on the side of caution and disabled the human review part of the process meant to identify videos falsely mislabeled by the platform’s A.I. system as a violation of terms. The change is still ongoing, and YouTubers who believe their videos were miscategorized are encouraged to file for reinstatement. Some search functions also remain disabled.

While YouTube didn’t share exact numbers of the uploads, Facebook says it removed 1.5 million. About 80 percent of those, 1.2 million, were blocked at upload before ever making it onto the platform, while another 300,000 were removed within the first 24 hours after posting. The live video saw fewer than 200 views, the company said, and the video uploaded by the shooter saw around 400 views before being removed. No users reported the video until 29 minutes after the shooter started livestreaming. The platform deleted the suspected shooter’s account on both Facebook and Instagram and banned the user.

Some of social media’s past efforts to keep violence and hate out worked — like the 1.3 million videos that never uploaded to Facebook. YouTube’s previous mistakes that allowed violent videos into the search suggestions didn’t appear to come into play, and users searching for the incident were redirected towards news coverage of the incident instead.

But as the videos numbering in the hundreds of thousands show, the A.I. designed to recognize offending videos isn’t foolproof. Many networks use what’s called hashing to prevent mass uploads by recognizing when the same video is uploaded more than once. But according to The Washington Post, some users have been able to bypass the hashing by shortening the video, adding logos, or even using an effect that made the real-life event look like a video game. While the original video was removed, the networks struggled with “remixes” of the livestream.

Facebook expanded its hashing technology to try to catch more variations on the video, adding audio hashing to the process. Networks that are part of with the Global Internet Forum to Counter Terrorism — which includes Facebook, YouTube, Twitter, and Microsoft — added variations of the video to a database, allowing other networks to prevent the same uploads. Facebook says the group together added around 800 variations of the video to the database.

“This was a tragedy that was almost designed for the purpose of going viral. We’ve made progress, but that doesn’t mean we don’t have a lot of work ahead of us, and this incident has shown that, especially in the case of more viral videos like this one, there’s more work to be done,” Mohan told The Washington Post.

Facebook didn’t comment on why 20 percent of videos that went live on the network weren’t caught at upload. “We continue to work around the clock to remove violating content using a combination of technology and people,” Facebook New Zealand representative Mia Garlick stated in a tweet. “Out of respect for the people affected by this tragedy and the concerns of local authorities, we’re also removing all edited versions of the video that do not show graphic content.”

Reddit and Twitter also removed related content from their platforms, but didn’t share related statistics. “We are continuously monitoring and removing any content that depicts the tragedy, and will continue to do so in line with the Twitter rules,” Twitter Safety tweeted. “We are also in close coordination with New Zealand law enforcement to help in their investigation.”

Updated on March 19 with additional details from Facebook.

Editors' Recommendations

Meta found over 400 mobile apps ‘designed to steal’ Facebook logins
Social media mobile apps on a smartphone screen, all on a textured gray fabric background.

If you frequently use your Facebook login to sign into new mobile apps you've installed, you may want to pay attention to Meta's latest announcement.

On Friday, Facebook's parent company Meta published a blog post written by its Director of Threat Disruption David Agranovich, and Ryan Victory, a Malware Discovery and Detection engineer at Meta. The post detailed Meta's discovery of over 400 mobile apps "that target people across the internet to steal their Facebook login information." Essentially, Meta found hundreds of mobile apps that were "designed to steal"  the login information of Facebook users by having those users log in to these apps with their Facebook login information.

Read more
Facebook’s new controls offer more customization of your Feed
A smartphone with the Facebook app icon on it all on a white marble background.

Facebook isn't likely to stop recommending posts in your Feed anytime soon, but it is offering a few options for controlling the content you see there.

On Wednesday, Facebook parent company Meta announced that the social networking platform is offering two more ways to customize your feed: by selecting "Show more" or "Show less" on individual posts, and by adjusting new settings in Feed Preferences.

Read more
New malware can steal your credit card details — and it’s spreading fast
An individual surrounded by several computers typing on a laptop.

A new, highly dangerous malware called "Erbium" has been making the rounds over the last couple of months, and it's highly likely that it will spread to new channels.

Erbium is an information-stealing tool that targets passwords, credit card information, cookies, cryptocurrency wallets, and more. Unfortunately, it's widely available, which means that it could be used in new ways in the future.

Read more