Skip to main content

Real-time search is Twitter’s priority, even if mere humans are used to power it

twitter
Image used with permission by copyright holder

Twitter’s priories are becoming clearer. The platform has long being tip-toeing into real-time search, in a blog post today made it clear how big of a priority this is becoming and the investment the team is making in developing this feature.

Since the shelf life of a trend is just mere hours (if that), engineers have to get creative with how to recognize and surface emerging trends in its search results. With this in mind, the social network’s updated search engine will now use crowdsourced human feedback to power its results. The problem with Twitter’s search engine before this change was that it couldn’t respond fast enough. When Obama’s re-election photo was first tweeted, or when the breaking story about Osama Bin Laden’s death first hit, its algorithm wasn’t able to recognize that trending hashtags relevant to these incidents in their initial occurrence on the site. 

Recommended Videos

For instance, in an example described by Twitter data scientist Edwin Chen and Senior Software Engineer Alpa Jain, the moment that Mitt Romney infamously said “binders full of women” the Internet-connected world flocked to Twitter to find out more about the phrase. Searching for anything relevant to “binders full of women” on Twitter before Romney’s remark would have displayed results about “binders” and “women.” But immediately after Romney’s flub, the meaning of “binders full of women” had instantly changed. Unfortunately Twitter’s engine isn’t able to recognize this change in context quickly enough when it originally happens. 

To solve this contextual problem, Twitter engineers turned to not only machines but also to humans.

Simply put, there are three steps to how this new human-powered system works. The first step is to recognize trending topics. Twitter does this by identifying spikes in traffic around certain search keywords using its Storm system, which mines and makes sense of the deluge of information in real-time to recognize trending topics.

Now the next step is where it gets interesting: Twitter sends topics that have spiked to “human evaluators” to provide more details about the subject matter, including photos, whether it refers to a person or event, and the type of category the topic would fall under. And all this is done using Amazon’s Mechanical Turk – a crowdsource platform for programmers to call on the help of human beings to perform tasks that an algorithm wouldn’t be capable of completing. Twitter’s “evaluators,” though, the blog post says, are custom Mechanical Turk judges and not just an everyday Joe. In fact the judges are from around the world (ensuring that there’s always someone up at any hour) and paid on a full-time basis to identify these trends. These judges have even set up their own network to collaborate and discuss a task with one another. In the hands of the anonymous select few, it’s a heavy responsibility that these Twitter-employed fact finders are holding.

Finally, Twitter teaches its engine the context of the word for posterity. Now if you search for “binders full of women” or any variation of that, you’ll see tweets about politics, Obama, Romney, and of course the phrase itself. What the blog post also adds is that this human powered contextual topic identification system will help with better targeting ads. To use the same example, if you search for “binders full of women,” ads that are relevant (or remotely relevant) to politics would then show up instead of ads for back-to-school binders.

Using human beings to identify trends might sound like a step backwards, but seeing as how the technology in this area has yet to catch up, a human-powered solution is really the most efficient strategy for engineers, especially given the amount of content users are increasingly flooding Twitter with. For everyone else, now you can be rest assured that Twitter is probably now the fastest (and relatively reliable) real-time social search engine for current events.

Topics
Francis Bea
Former Digital Trends Contributor
Francis got his first taste of the tech industry in a failed attempt at a startup during his time as a student at the…
How to download Instagram photos for free
Instagram app running on the Samsung Galaxy Z Flip 5.

Instagram is amazing, and many of us use it as a record of our lives — uploading the best bits of our trips, adventures, and notable moments. But sometimes you can lose the original files of those moments, leaving the Instagram copy as the only available one . While you may be happy to leave it up there, it's a lot more convenient to have another version of it downloaded onto your phone or computer. While downloading directly from Instagram can be tricky, there are ways around it. Here are a few easy ways to download Instagram photos.

Read more
How to get verified on Instagram Threads
A verified account on Instagram Threads.

Like Twitter and Instagram, Instagram Threads allows users to become verified to confirm their identities and access some exclusive features. Similar to Twitter's verification process, you'll need to pay a monthly fee to be verified on Threads, so keep that price in mind as you get your verified Threads account set up.

So, without further ado, here's how to get verified on Instagram Threads in a few straightforward steps.

Read more
How to remove location data from your iPhone photos
How to transfer photos from an iPhone to an iPhone

We all love making memories, and a great way to collect those memories is to take a quick snap of a gorgeous landscape, a party in full swing, or a particularly incredible meal. The Apple iPhone now also adds a location to your pictures, meaning it can collate those images together into a location-themed album, or show you all the shots you've taken in a specific location. It's a fun little addition, and it's one that adds a lot of personality to the Photos app.

Read more