How Facebook, Twitter, and Bing are giving researchers the perfect guinea pig (you)

Microsoft lab opening

Vast amounts of data and insane computational power are transforming what we know about how people interact with each other – but beyond that, social scientists have no idea what the hell is going on.

That’s the mad state of modern sociology, according to an array of researchers speaking Monday during an open house at Microsoft Research’s New York offices. While disciplines such as anthropology, communication, media studies, psychology, and sociology have seen markedly little improvement over the past hundred years, experts in these fields now see social networks like Facebook and Twitter as their great hope.

“It’s possible to look at the interactions of a billion people these days — we call it Facebook.”

Duncan Watts, a principal researcher at Microsoft Research, explained sociology’s stunted growth simply: Observing the interactions of hundreds of millions of people isn’t easy, and doing experiments on that scale was simply inconceivable. Until a few years ago, that is.

“It’s possible to look at the interactions of a billion people these days – we call it Facebook,” Watts said. “We have the potential for a revolution … not just in society but in our ability to study society.”

He likened it to the particle collider – a heckuva comparison to be sure, but for a research field that hasn’t been tweaked in a century, the advent of enormous databases that researchers can sift through and do experiments on in real time is a game-changer. Field research in many of these disciplines previously involved months or years of work. Databases popularly cited include the U.S. Census and Gallop Polls.

Today, Watts said researchers have “virtual labs” that allow them to test a hypothesis in days or even hours, using a hundred thousand or even a million test subjects. And it’s enabling all sorts of neat new studies.

Take for example a recent finding by Principal Researcher Kate Crawford, who studies crisis informatics (“they’re paying us by the buzzword,” joked her colleague Justin Rao). She studied Twitter following Hurricane Sandy to see how people’s relationship to privacy information changes during a crisis.

Microsoft lab opening 4“Yes, at the aggregate level there is a marked change,” she said, teasing the conclusion to a forthcoming research paper. “Simply put, when people are suffering, the amount of effort they’re going to put into protecting their data is much lower.”

Others are looking at how disease spreads, what leads to moral corruption, what makes us happy. But for a society wary of NSA prying and companies aggregating their lives for commercial purposes, the potential for researchers peering at their dirty laundry and drawing conclusions is scary.

After all, most people can’t even do basic probabilities; how are they supposed to understand that researchers aren’t piling on and further eroding their privacy? Social scientists say they’re aware of the issue.

“A lot of the data [scientists] are collecting is being increasingly interfered with by the companies that collect it.”

“We, as a scientific community, don’t even have a way of giving people even an inkling of what we’re talking about. We have a challenge as a community, and if we don’t work to get it right, we’re all guilty of something we don’t want to be guilty of,” said Dan Huttenlocher, dean and vice provost of Cornell Tech.

There’s an extra ingredient in the complex stew: Companies like Facebook and Bing aren’t creating the ideal raw data stream you might think they are.

“A lot of the data [scientists] are collecting is being increasingly interfered with by the companies that collect it,” Watts said, a problem he termed algorithmic conflict. Think of a Facebook post you make that the company only shares with a select group of people it thinks will be most likely to like it, rather than the pool of followers at large.

A 2012 study from Microsoft of Xbox user’s voting plans underscores the problem, failing to predict results. David Rothshchild, an economist with Microsoft wearing the requisite bow tie, compared it to an election-prediction fail from Literary Digest in 1936. Back then, the magazine’s flawed polling methods skewed toward Americans with high incomes, botching its predictions and eventually causing the magazine to fold. Likewise, Microsoft’s polling of Xbox users skewed toward younger gamers, largely male, and not the population at large.

Microsoft lab opening 3

“There’s been a lot of changes in the last 75 years, including the Internet and computers,” he joked. Yet traditional polling hasn’t changed, and we haven’t overcome issues we faced way back then.

Rao and Rothschild proposed yet a new type of database, which Rao called “medium data” rather than big data or small data. (He may or may not have been serious about the idea; Rao joked before beginning that “we didn’t have anything planned, so here goes.”)

Medium data would take the best parts of big data and little data, to get the best of both worlds: Giant data sets that can be mined for interrelations quickly and accurately. It could be the solution all of these social scientists are hoping for; or it could be a big joke.

“And again, you can forget this all after the talk if you want,” he concluded.

Social science is clearly being shaken up, likely in a good way, by the advent of machine learning and the vast data analysis it brings with it. It’s a long-overdue improvement. And one that clearly requires work.

“I really think this is a challenge we owe ourselves to spend some time thinking about,” Huttenlocher.


The history of Battle Royale: From mod to worldwide phenomenon

Battle royale games like PlayerUnknown's Battlegrounds’ and Fortnite have become the biggest trend in video games. The genre is also pushing the envelope in streaming and eSports in a way that might hint at the future of the industry.

Social media use increases depression and anxiety, experiment shows

A study has shown for the first time a causal link between social media use and lower rates of well-being. Students who limited their social media usage to 30 minutes a day showed significant decreases in anxiety and fear of missing out.

Data stolen from includes partial SSNs and immigration status

Around 75,000 users have had their user data stolen from government site, including information on their immigration status, whether they were pregnant, and partial social security numbers.
Movies & TV

The best shows on Netflix, from 'The Haunting of Hill House’ to ‘The Good Place’

Looking for a new show to binge? Lucky for you, we've curated a list of the best shows on Netflix, whether you're a fan of outlandish anime, dramatic period pieces, or shows that leave you questioning what lies beyond.
Emerging Tech

Computers will soon outsmart us. Does that make an A.I. rebellion inevitable?

At this point, the question isn’t so much “if” AI will ever surpass humans in terms of thinking abilities, it’s “when.” What happens when we reach that point? Charles J. Simon attempts to answer that question in his upcoming…

What is Android fragmentation, and can Google ever fix it?

Fragmentation on the Android platform has long been criticized as a problem for security, consistency, and app development. We take a look at Google’s attempted fixes and ask if it can do more.

Looking Glass owners will soon be able to get more holograms on Vimeo

We're inching closer to recreating the iconic scene in Star Wars of Princess Leia calling out to Obi-Wan for help. A Brooklyn company has created the Looking Glass, a holographic display that lets you see 3D content without a headset.

The MacBook of 2021 could kiss the keyboard, and Intel, goodbye

Apple announced a new MacBook Air, which brings a fairly basic update to the line. But if you follow the rumors, there's a much bigger sea change happening. Based on patents, rumors, and pure imagination, here's what we think it'll be like.

Infiniti is using Formula One racing to hone its hybrid skills

Infiniti is partnering with the Renault Sport Formula One team, and not just to put sponsor stickers on the race cars. The luxury brand claims its engineers are working with their F1 counterparts to develop future hybrid tech.

Blizzard Co-Founder Allen Adham says ‘we have not forgotten’ core Diablo fans

Blizzard fans are skeptical of 'Diablo Immortal,' but company co-founder Allen Adham is confident players will like it when they play it. He told us that multiple Diablo projects are underway and hinted at end-game content in 'Diablo…

Privacy is becoming obsolete, but not everyone thinks you should fear its demise

As technologies like Alexa and Siri that require more information about us continue to develop, is privacy going to fall at the wayside, or can we take back control of our data to retain our privacy?
Emerging Tech

To make more room for livestock, the Dutch will moove cows to a floating farm

A Dutch company is developing a floating dairy farm, which they hope to use as a proof-of-concept for future agricultural systems. The farm will use automated cleaning and milking robots, while recycling waste into fertilizer.
Emerging Tech

Driverless cars can only take you so far. This is how smarter cities are being built

Companies and municipalities came together at the annual Smart Mobility Summit, to share new technologies for building smarter cities. They also offered insight into the challenge cities face.

As Amazon turns up the volume on streaming, Spotify should shudder

Multiple players are all looking to capitalize on the popularity of streaming, but it has thus far proved nearly impossible to make a profit. Could major tech companies like Amazon be primed for a streaming take-over?