Skip to main content

Real-time search is Twitter’s priority, even if mere humans are used to power it

twitter
Image used with permission by copyright holder

Twitter’s priories are becoming clearer. The platform has long being tip-toeing into real-time search, in a blog post today made it clear how big of a priority this is becoming and the investment the team is making in developing this feature.

Since the shelf life of a trend is just mere hours (if that), engineers have to get creative with how to recognize and surface emerging trends in its search results. With this in mind, the social network’s updated search engine will now use crowdsourced human feedback to power its results. The problem with Twitter’s search engine before this change was that it couldn’t respond fast enough. When Obama’s re-election photo was first tweeted, or when the breaking story about Osama Bin Laden’s death first hit, its algorithm wasn’t able to recognize that trending hashtags relevant to these incidents in their initial occurrence on the site. 

For instance, in an example described by Twitter data scientist Edwin Chen and Senior Software Engineer Alpa Jain, the moment that Mitt Romney infamously said “binders full of women” the Internet-connected world flocked to Twitter to find out more about the phrase. Searching for anything relevant to “binders full of women” on Twitter before Romney’s remark would have displayed results about “binders” and “women.” But immediately after Romney’s flub, the meaning of “binders full of women” had instantly changed. Unfortunately Twitter’s engine isn’t able to recognize this change in context quickly enough when it originally happens. 

To solve this contextual problem, Twitter engineers turned to not only machines but also to humans.

Simply put, there are three steps to how this new human-powered system works. The first step is to recognize trending topics. Twitter does this by identifying spikes in traffic around certain search keywords using its Storm system, which mines and makes sense of the deluge of information in real-time to recognize trending topics.

Now the next step is where it gets interesting: Twitter sends topics that have spiked to “human evaluators” to provide more details about the subject matter, including photos, whether it refers to a person or event, and the type of category the topic would fall under. And all this is done using Amazon’s Mechanical Turk – a crowdsource platform for programmers to call on the help of human beings to perform tasks that an algorithm wouldn’t be capable of completing. Twitter’s “evaluators,” though, the blog post says, are custom Mechanical Turk judges and not just an everyday Joe. In fact the judges are from around the world (ensuring that there’s always someone up at any hour) and paid on a full-time basis to identify these trends. These judges have even set up their own network to collaborate and discuss a task with one another. In the hands of the anonymous select few, it’s a heavy responsibility that these Twitter-employed fact finders are holding.

Finally, Twitter teaches its engine the context of the word for posterity. Now if you search for “binders full of women” or any variation of that, you’ll see tweets about politics, Obama, Romney, and of course the phrase itself. What the blog post also adds is that this human powered contextual topic identification system will help with better targeting ads. To use the same example, if you search for “binders full of women,” ads that are relevant (or remotely relevant) to politics would then show up instead of ads for back-to-school binders.

Using human beings to identify trends might sound like a step backwards, but seeing as how the technology in this area has yet to catch up, a human-powered solution is really the most efficient strategy for engineers, especially given the amount of content users are increasingly flooding Twitter with. For everyone else, now you can be rest assured that Twitter is probably now the fastest (and relatively reliable) real-time social search engine for current events.

Editors' Recommendations

Topics
Francis Bea
Former Digital Trends Contributor
Francis got his first taste of the tech industry in a failed attempt at a startup during his time as a student at the…
X rival Threads could be about to get millions of more users
Instagram Threads app.

Threads -- Meta’s rival to X, formerly Twitter -- has just launched in the European Union (EU), a market with nearly half a billion people.

The app launched in the U.S. to much fanfare in July, with Meta hoping to attract X users disillusioned with the turbulence on the platform since Elon Musk acquired it for $44 billion 14 months ago.

Read more
X (formerly Twitter) returns after global outage
A white X on a black background, which could be Twitter's new logo.

X, formerly known as Twitter, went down for about 90 minutes for users worldwide early on Thursday ET.

Anyone opening the social media app across all platforms was met with a blank timeline. On desktop, users saw a message that simply read, "Welcome to X," while on mobile the app showed suggestions for accounts to follow.

Read more
How to create multiple profiles on a Facebook account
A series of social media app icons on a colorful smartphone screen.

Facebook (and, by extension, Meta) are particular in the way that they allow users to create accounts and interact with their platform. Being the opposite of the typical anonymous service, Facebook sticks to the rule of one account per one person. However, Facebook allows its users to create multiple profiles that are all linked to one main Facebook account.

In much the same way as Japanese philosophy tells us we have three faces — one to show the world, one to show family, and one to show no one but ourselves — these profiles allow us to put a different 'face' out to different aspects or hobbies. One profile can keep tabs on your friends, while another goes hardcore into networking and selling tech on Facebook Marketplace.

Read more