What are ‘Dark Google’ and ‘Dark Social?’ A webmaster’s mortal enemies, with no fix in sight

dark google dark socialHere’s a puzzling situation: An up-and-coming website looks at how people find it online – but things aren’t adding up.  A big chunk of Web traffic seemingly appears out of nowhere, befuddling the people trying to build an audience. After all, it’s hard to grow a fan base when you don’t know where your existing viewers come from. 

That might seem like an unusual scenario in the age of SEO and constant analytics, but it’s not. The Atlantic is an enormously popular online portal with staff dedicated to poring over traffic, yet one quarter of their Web traffic comes from this unknown source. Buzzfeed also pointed out decreases in traditional traffic sources and an increase in unspecific traffic. And two interconnected culprits are responsible for this analytics-defying traffic: Dark social and dark Google. 

But in 2011 things changed, and Google started withholding around 50 percent of this information.

Dark social and dark Google are something that anyone with a website – even a Tumblr or WordPress blog they want to take off – should care about, because it makes it impossible to accurately gauge your audience, and it also makes it harder to figure out why certain content succeeds and other content falls flat. It’s like an analytics night terror – and the problem may continue to grow as more people use mobile devices to access the Web, since iPads and iPhones throw the most data into the dark.

So what are dark Google and dark social?

Google used to report where traffic came from through its search engine very accurately; withheld information was rare. But in 2011 things changed, and Google started withholding around 50 percent of this information from Web publishers. This withheld information is dark Google.

Dark social is a term coined by The Atlantic’s tech writer Alexis Madrigal to explain the Web traffic that comes from unspecified, mysterious social shares. Dark social accounts for all the people who find articles through sharing, but not on traditional sharing sources like Facebook, Twitter, Reddit, and the other big sources for finding links from others. Dark social is not a little portion of social links; in 2012, Madrigal said it accounted for 69 percent of social sharing. That’s a whole lot of link-swapping unaccounted for by Web analysts, and a whole lot of missed opportunities for people trying to measure their audience.

Rosetta SEO director Stephen Pitts says dark social is the offspring of dark Google. “It’s the same root cause as dark Google,” he explains. But while dark Google is hidden search traffic, dark social is hidden traffic that stems from people sharing links.

The shared cause is the traffic chaos they create. Pitts describes dark social and dark Google as a data referral problem at its core. The information websites need to analyze their traffic is simply not getting passed on anymore. “There is a function that is normally handled by browsers that is a meta-referrer for information. The meta-referrer takes the keyword if it’s coming from a paid or organic search in, say, Google, and sends that to an analytics package. In analytics, it would tell you who the referrer was and the keyword. Now, in this circumstance, what happened is last year Google Search decided to stop providing that as a function because it’s handled by most browsers. So now they put the onus on passing that data to the browsers.” 

And now that browsers are in charge, things are falling out of whack. “That wasn’t a problem until Apple, with the release of the iPhone 5 and iOS 6, decided they were trimming down all the data being requested in their browsers. Apple stopped supporting meta-referrer data. So if any iOS device is used, whether it be to instant message, text message, to use an app that’s sending you to a website, it won’t pass the meta-referrer information. So the analytics program thinks its direct traffic.” So there’s no simple way to measure how much is dark Google and how much is dark social; both types of traffic just end up unmeasured.

Even though this creates a giant headache for people trying to figure out their audiences, don’t expect a change.

Not every mobile browser is a problem, though. Pitts uses the Facebook app as an example of a service that still passes data along. “The Facebook app is a little different. If I’m on Facebook on my iPhone and I click to an article on the Web, I’m on the Facebook browser, which still passes the meta-referrer data. The browser functionality on the Facebook app is still providing the information.” That’s not surprising, considering Facebook’s vested interest in collecting data on its users. It wouldn’t want to alienate potential ad-buyers by refusing to confer information.

Apple is by and large the biggest culprit when it comes to not passing along the necessary meta-referrers. “If you use Chrome on your iPhone, the data still gets passed. It’s just if you use Safari [it isn’t].” And Pitts hadn’t heard of similar problems on Android. “It’s mostly iOS. When you’re on an Android device you’re almost always defaulting to a Chrome browser – and it’s mostly Safari that’s the issue.”

But Firefox might also cause problems later on. “They’re talking about trimming everything except first-party cookies,which would in essence do the same thing as iOS.”

So Google assumes that browsers will take responsibility for passing along data, but Apple hasn’t taken up that job with Safari. And thus, marketers, webmasters, and other groups concerned with analytics lose valuable information – especially considering the amount of iPhones and other iOS products out there.

Even though this creates a giant headache for people trying to figure out their audiences, don’t expect a change.”Neither Google nor Apple have incentive to make a change,” Pitts says. “The people who are losing this insight are the marketers and the website owners, which is disappointing because we can’t optimize our sites to improve user experience if we can’t use the data.”

“We see it less likely to happen on financial services and B2B sites, while it’s more likely to happen on retail and hospitality sites,” Pitts explains, noting that people tend to online shop on their phones, while they often wait to do things like refinance their homes when they’re on a traditional, more secure desktop. That doesn’t mean it never happens, though. “It’s amazing to me that people go through tedious, 18 page documents on tiny screens. But people do it. It might be their only device that’s connected.”

Unless people analyzing traffic figure out a workaround, as long as people continue to use Safari and other browsers that don’t send meta-referrers along, this problem will persist and continue to stymy those who want to know their audience.

Is there any benefit for users? Well, dark Google and dark social confound attempts by marketers to figure out ways to target ads, so if you really hate seeing targeted ads, you may take solace in the fact that both Google and Apple are withholding information. Then again, if you like using retail sites that customize their services based on your past preferences, both dark Google and dark social make it harder for tailored user experiences. And anyone with a website they want to drive traffic to – which is a considerable number of you out there – will continue in the frustration of never really knowing all your readers. 


Think iPhones can’t get viruses? Our expert explains why it could happen

If your iPhone has been acting strangely, then you may be concerned about the possibility it is infected with a virus or some malware. We take a look at just how likely that is and explain why iOS is considered relatively safe.

Switch up your Reddit routine with these interesting, inspiring, and zany subs

So you've just joined the wonderful world of Reddit and want to explore it. But with so many subreddits to choose from, exploring them can be overwhelming. Here are some of the best subreddits to get you started.
Emerging Tech

Gorgeous image of the Cosmic Bat nebula leaves us starry-eyed

The "Cosmic Bat" nebula has been captured in beautiful detail by the European Southern Observatory. Formally known as NGC 1788, the nebula is two thousand light-years away in a dark corner of the Orion constellation.

The first Android Q beta is available. Here's what we know about it

Android Pie recently rolled out, but it's already time to look ahead to Android Q, the next version of Google's mobile OS. We've seen a number of rumors and leaks come out about the operating system, and now it's available for beta testing.

Make a GIF of your favorite YouTube video with these great tools

Making a GIF from a YouTube video is easier today than ever, but choosing the right tool for the job isn't always so simple. In this guide, we'll teach you how to make a GIF from a YouTube video with our two favorite online tools.
Social Media

Twitter takes a cue from Instagram and Snapchat with new quick-swipe camera

Twitter is giving the "what's happening" treatment to photos and video by allowing users to access the in-app camera fast enough to catch and share the moment. The new Twitter camera is now accessible with a swipe.
Social Media

Yep, it’s not just you. Facebook, Instagram, and WhatsApp are down for many

Facebook's family of apps has been suffering issues for much of the day. Instagram, WhatsApp, Facebook Messenger, and Facebook itself have been out of action for users around the world, with the company scrambling to sort it out.
Social Media

Facebook may soon let you watch live TV with friends in Watch Party

Facebook Watch Party is designed to allow friends to watch together, even when they can't be in the same physical space. Now, that feature could be expanding to include live TV. Facebook announced a test of the feature, starting with live…
Social Media

Federal investigation digs into Facebook’s data-sharing deals

Facebook confirmed it is cooperating with a federal criminal investigation. According to a report, the company is under investigation for sharing user data with smartphone and tablet companies.
Social Media

Facebook explains its worst outage as 3 million users head to Telegram

Facebook, if you didn't already know it, suffered a bit of an issue on Wednesday, March 13. An issue that took down not only its social networking site, but also Instagram, WhatsApp, and Messenger. On Thursday it offered an explanation.

Snapchat could soon let you play games in between your selfies

If a new report is accurate, Snapchat will be getting an integrated gaming platform in April. The platform will feature mobile games form third-party developers, and one publisher is already signed on.
Social Media

Twitter is testing a handy subscription feature for following threads

Twitter has recently started testing a feature that lets you subscribe to a thread so that you’ll no longer need to like a comment or post to it yourself in order to receive notifications of new contributions.
Social Media

Your Google+ public content will remain viewable on the web, if you want it to

Google's failed social network — Google+ — will soon be wiped from the internet, but there's a team of volunteers working right now to save its public content for the Internet Archive.

There’s more space on MySpace after ‘accidental’ wipe of 50 million songs

MySpace is no longer a safe refuge for music and media produced in the 2000s. It said that almost any artistic content uploaded to the site between 2003 and 2015 may have been lost as part of a server migration last year.