What are ‘Dark Google’ and ‘Dark Social?’ A webmaster’s mortal enemies, with no fix in sight

dark google dark socialHere’s a puzzling situation: An up-and-coming website looks at how people find it online – but things aren’t adding up.  A big chunk of Web traffic seemingly appears out of nowhere, befuddling the people trying to build an audience. After all, it’s hard to grow a fan base when you don’t know where your existing viewers come from. 

That might seem like an unusual scenario in the age of SEO and constant analytics, but it’s not. The Atlantic is an enormously popular online portal with staff dedicated to poring over traffic, yet one quarter of their Web traffic comes from this unknown source. Buzzfeed also pointed out decreases in traditional traffic sources and an increase in unspecific traffic. And two interconnected culprits are responsible for this analytics-defying traffic: Dark social and dark Google. 

But in 2011 things changed, and Google started withholding around 50 percent of this information.

Dark social and dark Google are something that anyone with a website – even a Tumblr or WordPress blog they want to take off – should care about, because it makes it impossible to accurately gauge your audience, and it also makes it harder to figure out why certain content succeeds and other content falls flat. It’s like an analytics night terror – and the problem may continue to grow as more people use mobile devices to access the Web, since iPads and iPhones throw the most data into the dark.

So what are dark Google and dark social?

Google used to report where traffic came from through its search engine very accurately; withheld information was rare. But in 2011 things changed, and Google started withholding around 50 percent of this information from Web publishers. This withheld information is dark Google.

Dark social is a term coined by The Atlantic’s tech writer Alexis Madrigal to explain the Web traffic that comes from unspecified, mysterious social shares. Dark social accounts for all the people who find articles through sharing, but not on traditional sharing sources like Facebook, Twitter, Reddit, and the other big sources for finding links from others. Dark social is not a little portion of social links; in 2012, Madrigal said it accounted for 69 percent of social sharing. That’s a whole lot of link-swapping unaccounted for by Web analysts, and a whole lot of missed opportunities for people trying to measure their audience.

Rosetta SEO director Stephen Pitts says dark social is the offspring of dark Google. “It’s the same root cause as dark Google,” he explains. But while dark Google is hidden search traffic, dark social is hidden traffic that stems from people sharing links.

The shared cause is the traffic chaos they create. Pitts describes dark social and dark Google as a data referral problem at its core. The information websites need to analyze their traffic is simply not getting passed on anymore. “There is a function that is normally handled by browsers that is a meta-referrer for information. The meta-referrer takes the keyword if it’s coming from a paid or organic search in, say, Google, and sends that to an analytics package. In analytics, it would tell you who the referrer was and the keyword. Now, in this circumstance, what happened is last year Google Search decided to stop providing that as a function because it’s handled by most browsers. So now they put the onus on passing that data to the browsers.” 

And now that browsers are in charge, things are falling out of whack. “That wasn’t a problem until Apple, with the release of the iPhone 5 and iOS 6, decided they were trimming down all the data being requested in their browsers. Apple stopped supporting meta-referrer data. So if any iOS device is used, whether it be to instant message, text message, to use an app that’s sending you to a website, it won’t pass the meta-referrer information. So the analytics program thinks its direct traffic.” So there’s no simple way to measure how much is dark Google and how much is dark social; both types of traffic just end up unmeasured.

Even though this creates a giant headache for people trying to figure out their audiences, don’t expect a change.

Not every mobile browser is a problem, though. Pitts uses the Facebook app as an example of a service that still passes data along. “The Facebook app is a little different. If I’m on Facebook on my iPhone and I click to an article on the Web, I’m on the Facebook browser, which still passes the meta-referrer data. The browser functionality on the Facebook app is still providing the information.” That’s not surprising, considering Facebook’s vested interest in collecting data on its users. It wouldn’t want to alienate potential ad-buyers by refusing to confer information.

Apple is by and large the biggest culprit when it comes to not passing along the necessary meta-referrers. “If you use Chrome on your iPhone, the data still gets passed. It’s just if you use Safari [it isn’t].” And Pitts hadn’t heard of similar problems on Android. “It’s mostly iOS. When you’re on an Android device you’re almost always defaulting to a Chrome browser – and it’s mostly Safari that’s the issue.”

But Firefox might also cause problems later on. “They’re talking about trimming everything except first-party cookies,which would in essence do the same thing as iOS.”

So Google assumes that browsers will take responsibility for passing along data, but Apple hasn’t taken up that job with Safari. And thus, marketers, webmasters, and other groups concerned with analytics lose valuable information – especially considering the amount of iPhones and other iOS products out there.

Even though this creates a giant headache for people trying to figure out their audiences, don’t expect a change.”Neither Google nor Apple have incentive to make a change,” Pitts says. “The people who are losing this insight are the marketers and the website owners, which is disappointing because we can’t optimize our sites to improve user experience if we can’t use the data.”

“We see it less likely to happen on financial services and B2B sites, while it’s more likely to happen on retail and hospitality sites,” Pitts explains, noting that people tend to online shop on their phones, while they often wait to do things like refinance their homes when they’re on a traditional, more secure desktop. That doesn’t mean it never happens, though. “It’s amazing to me that people go through tedious, 18 page documents on tiny screens. But people do it. It might be their only device that’s connected.”

Unless people analyzing traffic figure out a workaround, as long as people continue to use Safari and other browsers that don’t send meta-referrers along, this problem will persist and continue to stymy those who want to know their audience.

Is there any benefit for users? Well, dark Google and dark social confound attempts by marketers to figure out ways to target ads, so if you really hate seeing targeted ads, you may take solace in the fact that both Google and Apple are withholding information. Then again, if you like using retail sites that customize their services based on your past preferences, both dark Google and dark social make it harder for tailored user experiences. And anyone with a website they want to drive traffic to – which is a considerable number of you out there – will continue in the frustration of never really knowing all your readers. 

Movies & TV

‘Sopranos’ prequel ‘The Many Saints of Newark’ adds ‘The Punisher’ star

The upcoming movie The Many Saints of Newark, based on HBO's hit series The Sopranos, will serve as a prequel to the show. It will focus on Tony Soprano as a boy, as well as the race riots that rocked Newark in 1967.

‘Battlefield V’ adds Squad Conquest mode in ‘Lightning Strikes’ update

Electronic Arts and DICE have detailed what is included in the Battlefield V update Lightning Strikes. The update includes access to the mode Squad Conquest through the end of January.
Movies & TV

Best new shows and movies to stream: ‘Annihilation,’ ‘True Detective’ and more

Need something to watch this weekend? Check out our list of the best new shows and movies to stream right now. On the list this week: Annihilation, the start of a new season of True Detective, and more.
Movies & TV

The best new movie trailers: ‘Velvet Buzzsaw,’ ‘IO,’ Fyre Festival, and more

Everyone loves a good trailer, but keeping up with what's new isn't easy. That's why we round up the best ones for you. This week, it's trailers for a Fyre Festival documentary, as well as Velvet Buzzsaw and Iron Sky: The Coming Race.

Photoshop fail gives Aussie leader two left feet in official portrait

The Aussie prime minister doesn't have two left feet, despite an official photograph of the leader and his family suggesting otherwise. It was, of course, a Photoshop fail, and the embarrassing snafu was soon trending on Twitter.
Social Media

Instagram now lets you post to multiple accounts in one tap

Instagram for iPhone now lets you post to multiple accounts at the same time. It's not the regram feature that many users have been asking for, but it could prove useful for some users who manage more than one profile.
Social Media

No yolk! A photo of an egg has become the most-liked post on Instagram

Until this weekend, the most-liked post on Instagram was of Kylie Jenner's baby daughter, which has around 18 million likes. It's now been knocked off the top spot not by a stunning sunset or even a cute cat, but by an egg.
Social Media

Invite your friends — Facebook Events can now be shared to Stories

Facebook is testing a way to make plans with friends to attend an event -- through Stories. By sharing an event in Facebook Stories, users can message other friends interested in the event to make plans to attend together.
Social Media

A quick swipe will soon let you keep bingeing YouTube on mobile devices

The YouTube mobile app has a new, faster way to browse: Swiping. Once the update rolls out, users can swipe to go to the next (or previous) video in the recommended list, even while viewing in full screen.

Starting your very own vlog? Here are the best cameras to buy

Any camera that shoots video can be used to vlog, but a few models stand out from the crowd thanks to superior image quality, ergonomics, and usability. When it comes to putting your life on YouTube, here are the best cameras for the job.
Social Media

Twitter extends its new timeline feature to Android users

Twitter users with an Android device can now quickly switch between an algorithm-generated timeline and one that shows the most recent tweets first. The new feature landed for iPhone users last month.
Social Media

YouTube to crack down on dangerous stunts like the ‘Bird Box’ challenge

YouTube already bans content showing dangerous activities, but new rules published by the site go into greater detail regarding potentially harmful challenges and pranks, including certain blindfold- or laundry detergent-based stunts.
Social Media

Nearly 75 percent of U.S. users don’t realize Facebook tracks their interests

Did you know Facebook tracks your interests, including political and multicultural affiliations? According to a recent Pew study, 74 percent of adult users in the U.S. have no idea Facebook keeps a running list of your interests.