Skip to main content

Real-time search is Twitter’s priority, even if mere humans are used to power it


Twitter’s priories are becoming clearer. The platform has long being tip-toeing into real-time search, in a blog post today made it clear how big of a priority this is becoming and the investment the team is making in developing this feature.

Since the shelf life of a trend is just mere hours (if that), engineers have to get creative with how to recognize and surface emerging trends in its search results. With this in mind, the social network’s updated search engine will now use crowdsourced human feedback to power its results. The problem with Twitter’s search engine before this change was that it couldn’t respond fast enough. When Obama’s re-election photo was first tweeted, or when the breaking story about Osama Bin Laden’s death first hit, its algorithm wasn’t able to recognize that trending hashtags relevant to these incidents in their initial occurrence on the site. 

Related Videos

For instance, in an example described by Twitter data scientist Edwin Chen and Senior Software Engineer Alpa Jain, the moment that Mitt Romney infamously said “binders full of women” the Internet-connected world flocked to Twitter to find out more about the phrase. Searching for anything relevant to “binders full of women” on Twitter before Romney’s remark would have displayed results about “binders” and “women.” But immediately after Romney’s flub, the meaning of “binders full of women” had instantly changed. Unfortunately Twitter’s engine isn’t able to recognize this change in context quickly enough when it originally happens. 

To solve this contextual problem, Twitter engineers turned to not only machines but also to humans.

Simply put, there are three steps to how this new human-powered system works. The first step is to recognize trending topics. Twitter does this by identifying spikes in traffic around certain search keywords using its Storm system, which mines and makes sense of the deluge of information in real-time to recognize trending topics.

Now the next step is where it gets interesting: Twitter sends topics that have spiked to “human evaluators” to provide more details about the subject matter, including photos, whether it refers to a person or event, and the type of category the topic would fall under. And all this is done using Amazon’s Mechanical Turk – a crowdsource platform for programmers to call on the help of human beings to perform tasks that an algorithm wouldn’t be capable of completing. Twitter’s “evaluators,” though, the blog post says, are custom Mechanical Turk judges and not just an everyday Joe. In fact the judges are from around the world (ensuring that there’s always someone up at any hour) and paid on a full-time basis to identify these trends. These judges have even set up their own network to collaborate and discuss a task with one another. In the hands of the anonymous select few, it’s a heavy responsibility that these Twitter-employed fact finders are holding.

Finally, Twitter teaches its engine the context of the word for posterity. Now if you search for “binders full of women” or any variation of that, you’ll see tweets about politics, Obama, Romney, and of course the phrase itself. What the blog post also adds is that this human powered contextual topic identification system will help with better targeting ads. To use the same example, if you search for “binders full of women,” ads that are relevant (or remotely relevant) to politics would then show up instead of ads for back-to-school binders.

Using human beings to identify trends might sound like a step backwards, but seeing as how the technology in this area has yet to catch up, a human-powered solution is really the most efficient strategy for engineers, especially given the amount of content users are increasingly flooding Twitter with. For everyone else, now you can be rest assured that Twitter is probably now the fastest (and relatively reliable) real-time social search engine for current events.

Editors' Recommendations

Twitter’s handy new search tool makes it easier to find the perfect GIF
twitter new home timeline app

Twitter's testing of a dedicated GIF button first came to light earlier this month, and on Wednesday it announced it as an official feature.

The search button, unveiled by the company in a blog post, will make it easier for Twitter users to jazz up their tweets and direct messages with the entertaining image loops.

Read more
New York Times’ Marcus Mabry is joining the Twitter team to head Moments
turkish journalist jailed over tweets twitter pad

Twitter isn't wasting much time mourning the dearly departed. Just a week after announcing that it had fired eight percent of its workforce (some of whom found out in less than ideal ways), Twitter is charging full steam ahead and bringing aboard new talent, including New York Times editor at large Marcus Mabry, who will now be heading Twitter Moments.

Moments, a new feature announced in a Twitter blog post two weeks ago, is described as "the best of what’s happening on Twitter in an instant," and considering Mabry's background with the Times' “Watching” section, it seems that the former journalist is a perfect fit for the new team and the new role.

Read more
For real-time news, Facebook Notify could be the Twitter Moments killer
facebook notify paper hands on news 2

One of the most important competitions among social media platforms these days is the race to becoming the app providing one-stop features where users can get all their updates and information without ever leaving.  Facebook could become the platform that wins this race with its real-time news app, tentatively scheduled to be called “Notify,” according to a recent update from The Awl.

Although some of the larger online publications have successfully built substantial followings and readerships, most news apps are barely used and for the most part, are not competitive. Considering Facebook’s nearly 1.5 billion users alone, it seems a missed opportunity for not only Facebook but for online news sites as well.

Read more