Will researchers ever be able to wrap their arms around Twitter?


Even if you aren’t on Twitter, simply using the Internet (or watching TV) means you probably know about Twitter and its famous 140-character messages. Whether you consider it a fad, a social network, or a tool for photo or video sharing, Twitter is almost certainly in your face.

Twitter passed the 500-million-user milestone last year, meaning as many as one in eight people worldwide are on board. In theory, Twitter is a way to peer into the collective consciousness of the Internet — almost in real time. But the everyday Twitter experience remains surprisingly narrow: Most of us follow friends and family, a news source or two, and perhaps some celebrities or brands. But what about the big picture being painted by those hundreds of millions of users? What can we glean from our collective tweeting?

How tweet it is

A View from Above (TweetFlare) Vegas

Twitter itself doesn’t reveal much about its users or its own overall activity. Aside from self-congratulatory blog posts about events like the Super Bowl, Twitter merely highlights current hashtags and topics in its site and apps, and forgets them within minutes or hours. Unless your world revolves around something like #ExplainToMeWhy or boasting about follower counts, it’s not a deep look at Twitter’s overall spirit.

However, access to near real-time Twitter data has inspired efforts to look for patterns in tweet activity. An early example was Nikhil Bobb‘s A View from Above. It’s more of an aesthetic experiment than a utility, but it plots tweets with location data as flares on a black background: the more tweets in an area, the bigger and brighter the flare.

“The early inspiration was to approximate how an omnipotent being would see humanity from above,” Bobb wrote via email. “As I got more into the project, I found the digital cityscapes were a different and often more beautiful reflection of their geographic equivalents.”


Franck Ernewein’s TweetPing takes that idea one step further, plotting global tweets in real time against a map of the world. It’s an intriguing display — particularly when one realizes how few tweets seem to originate in Africa, let alone Iran or China. But watching TweetPing also reveals shortcomings. At almost any hour, hotspots like the U.S. east coast, western Europe, Tokyo, Indonesia, and Brazil’s southeast coast become incandescent, yet India remains almost dark. No one tweets in India? One might see more tweets from Greenland or Antarctica than Morocco, Bangladesh, or Madagascar. Sure, China blocks Twitter outright, but the real issue is that tweets don’t include GPS info. Last year, Semiocast estimated fewer than one percent of tweets included location data. The fascinating visuals of TweetPing (and A View From Above) reflect just a handful of Twitter users comfortable with (or capable of) sharing their location.

Revisit (Dell)

If a bird’s-eye view of Twitter clearly has limits, what about a topical view? Freelance information visualizer Moritz Stefaner took that on with Revisit, a Flash-based timeline that displays recent tweets matching search terms (like “iPhone” or “Seattle”). Revisit emphasizes conversations, highlighting replies and retweets. Where TweetPing offers a global view, Revisit is almost microscopic, intended for specific topics or events — but watching the ebb and flow can be just as fascinating.

Why Twitter’s hard to handle

Library of Congress (reading room)

These projects are just examples, but they demonstrate why Twitter is a tough problem: there’s just so much data. A comprehensive “Twitter tracker” might be impossible. Heck, Twitter itself hasn’t built one.

The highest-profile effort may be at the Library of Congress, which is creating a complete archive of every tweet going back to 2006. (Right now, it’s on tape, according to a recent white paper.) It’s ambitious: the Library of Congress started work in 2010, and is only now moving towards granting researchers access — public access is a long way off.

Why is it taking so long? If search engines like Google and Bing offer real-time Internet search, what’s hard about making Twitter searchable? It’s all about volume. Jamie de Guerre of the analytics firm Topsy recently noted Internet search engines might index 125 billion Web pages. In comparison, the Library of Congress’s Twitter archive is already over 170 billion tweets. That’s an apples-and-oranges comparison, but, in some ways, Twitter is a bigger problem than Internet search.

“Search engine companies have invested a lot of human and financial resources in building entire businesses on search. As a public institution, it is not practical for the Library to replicate that kind of investment,” wrote Gayle Osterberg, Director of Communications for the Library of Congress.

The private sector might help out with public access, but the details could be complicated: Think about advertising or charging for access. Still, there’s hope:

“We are looking at public-private partnerships to achieve some level of access,” noted Osterberg.

Tweet and consequences

Just as TweetPing uses the small number of tweets with location data, Twitter itself is not level ground. A 2011 study found U.S. Twitter users were disproportionately Caucasian, male, and urban. These demographics complicate any conclusions drawn from its data.

Nonetheless, the potential of the content and interaction tucked away in Twitter is staggering.

“Twitter provides us with an unprecedented, direct access to people’s ideas and thoughts at any given point in time,” noted Revisit creator Mortiz Stefaner. “I think we have only scratched the tip of the iceberg in what can be extracted.”

Meanwhile, the best and most accurate view of Twitter remains close to home in our own timelines — and it may be a long time before anyone else can make sense of us.


Car-branded phones need to make a U-turn if they ever want to impress

Your car and your smartphone are becoming one, yet smartphones branded or co-created by car companies are a problem. We look at the history, some examples of the best and worst, then share hopes for the future.
Movies & TV

The best shows on Netflix, from 'Haunting of Hill House’ to ‘Twilight Zone’

Looking for a new show to binge? Lucky for you, we've curated a list of the best shows on Netflix, whether you're a fan of outlandish anime, dramatic period pieces, or shows that leave you questioning what lies beyond.

The best free music download sites that are totally legal

Finding music that is both free and legal to download can be difficult. We've handpicked a selection of the best free music download sites for you to legally download your next favorite album.
Movies & TV

Tired of Netflix? Here's where to find free movies online, legally

We've spent countless hours digging around the web to find the best sites for streaming free movies online. Not only are all of these sites completely free to use, they're also completely legal and trustworthy.

Our favorite Windows apps will help you get the most out of your new PC

Not sure what apps you should be downloading for your newfangled Windows device? Here are the best Windows apps, whether you need something to speed up your machine or access your Netflix queue. Check out our categories and favorite picks.

5G’s arrival is transforming tech. Here’s everything you need to know to keep up

It has been years in the making, but 5G is finally becoming a reality. While 5G coverage is still extremely limited, expect to see it expand in 2019. Not sure what 5G even is? Here's everything you need to know.
Emerging Tech

Awesome Tech You Can’t Buy Yet: Folding canoes and ultra-fast water filters

Check out our roundup of the best new crowdfunding projects and product announcements that hit the web this week. You may not be able to buy this stuff yet, but it sure is fun to gawk!

Vanquish lag for good with the best routers for gaming

Finding the best routers for gaming is no easy task. With so many out there, how do you know which to pick? We've looked at the many options available and put together a list of our lag-free favorites.

Beddit Sleep Monitor 3.5 now available on the Apple Store

The Beddit Sleep Monitor 3.5 is now available on the Apple Store for $150. The sensor strip, which is only 2 millimeters thin, automatically tracks a wide array of sleep data when placed under the user's sheets.

Save up to $750 with the best smartphone deals for December 2018

Need a better phone but don't want to spend a fortune? It's never a bad time to score a new smartphone and save some cash. We rounded up the best smartphone deals available that can save you as much as $750.

Google to end support for Android devices running Ice Cream Sandwich

Anyone with an old phone that is still running Android 4.0 may want to look into upgrading their phone, as Google has announced that it will be ending support for this older version of Android.

Amazon knocks $30 off its Paperwhite ebook reader in limited-time deal

Amazon is running a couple of limited-time deals for its Paperwhite ebook reader. One offers a $30 discount, while the other throws in a pair of headphones and a free, extended Audible trial.

New Galaxy S10 leaks showcase display sizes, confirm headphone jack return

While we still may be months away from an announcement, there's no doubt about it: Samsung is working hard on its successor to the Galaxy S9. Here's everything we know about the upcoming Samsung Galaxy S10.

Android 9.0 updates to stretch into 2019 -- will your phone get a slice of Pie?

Android 9.0 Pie has been released. But is your phone getting Android 9.0 Pie, and if so, when? We've done the hard work and asked every device manufacturer to see when their devices would be getting the update.