Skip to main content

Twitter is using A.I. to ditch those awful auto-cropped photos

twitter auto crops improve with ai
Twindesign/123RF
The Twitter auto crop feature functions like a tweet’s character limit in order to keep images on the microblogging platform consistent with the rest of the feed — but now Twitter is getting better at those crops, thanks to artificial intelligence. Twitter is now rolling out a smarter auto crop based on neural networks, the company announced in a blog post on January 24.

The previous auto crop feature worked by using face detection to keep faces in the frame. When no faces were detected in the image, the software would simply crop the preview at the center, while a click on the image allowing users to see the entire shot. Twitter says the crop option without faces would often lead to awkward crops, while sometimes the software didn’t correctly identify faces.

Recommended Videos

To fix those awkwardly cropped previews, Twitter engineers used what’s called salient image maps to train a neural network. Salient maps use eye trackers to determine the areas of an image that most catch the viewer’s eye. Earlier research in the area showed that viewers tend to focus on faces, text, animals, objects, and areas with high contrast.

Please enable Javascript to view this content

Twitter used that earlier data to train the program to understand which areas of the image are the most important. Using that data, the program can recognize those features and make that auto crop in a place that will leave the most visual areas inside the crop.

But Twitter wasn’t done — while saliency software works well, it’s also slow, which would have prevented tweets from being posted in real time. To solve the awkward crops problem without a slowdown, Twitter refined the program again using two different techniques that improved the speed tenfold. The first trained a smaller network using that first good but slow program in order to speed up those crops. Next, the software engineers determined a number of visual points to map on each image, effectively removing the smaller, less important visual cues while keeping the largest areas intact.

Twitter Auto Crop
Before Image used with permission by copyright holder
Twitter Auto Crop
After Image used with permission by copyright holder

The resulting software allows images to post in real time, but with better crops. In a group of before and after pictures, Twitter shows images with faces that the earlier system wouldn’t detect properly cropped to face rather than feet. Other examples show images of objects that were cut out in the first program because they didn’t sit in the middle of the image, but were more appropriately cropped using the updated algorithms. Another example shows the program recognizing text and adjusting the crop to include a sign.

The updated cropping algorithm is already rolling out globally on both iOS and Android apps as well as Twitter.com.

Hillary K. Grigonis
Hillary never planned on becoming a photographer—and then she was handed a camera at her first writing job and she's been…
Thanks to Tapbots’ Ivory app, I’m finally ready to ditch Twitter for good
Profile displayed in Ivory app

Ever since Elon Musk took ownership of Twitter, it’s been one chaotic new thing after another. You literally cannot go a day (or a few days or even a week) without some stupid new change to the site — whether it’s about checkmarks for verified or Twitter Blue subscriber accounts, how links to other social networks are banned and then reversed, view counts on Tweets, or something else. I can’t keep up with every little thing that has happened since the beginning of November, and it feels like the spotlight is always on the toxicity of the site in general.

New Twitter alternatives have been popping up recently, but it seems that the most popular one continues to be Mastodon. I originally made a Mastodon account back in 2018 when it first launched, but it never clicked with me back then, and I eventually went back to Twitter. With the Musk mess, I tried going back to Mastodon, but again, it didn’t really click with me — until Tweetbot developer, Tapbots, revealed its next project: Ivory.
The significance of Tapbots and Tweetbot

Read more
Microsoft quits its creepy, emotion-reading A.I.
blonde woman with an expressionless face looks at camera while laser lights scan her features

Microsoft announced it will stop the development and distribution of controversial emotion-reading software as big tech companies pivot toward privacy and security. The company also says it will heavily restrict its own facial recognition platform.

Microsoft’s shift away from emotional recognition software is another sign of big tech’s growing prioritization of privacy. The company also admits there is little scientific evidence behind the technology.

Read more
Nvidia’s latest A.I. results prove that ARM is ready for the data center
Jensen Huang at GTX 2020.

Nvidia just published its latest MLPerf benchmark results, and they have are some big implications for the future of computing. In addition to maintaining a lead over other A.I. hardware -- which Nvidia has claimed for the last three batches of results -- the company showcased the power of ARM-based systems in the data center, with results nearly matching traditional x86 systems.

In the six tests MLPerf includes, ARM-based systems came within a few percentage points of x86 systems, with both using Nvidia A100 A.I. graphics cards. In one of the tests, the ARM-based system actually beat the x86 one, showcasing the advancements made in deploying different instruction sets in A.I. applications.

Read more