Breaking down ‘Big Data’ and Internet in the age of variety, volume, and velocity

State of the Web: Boiling down Big Data

Earlier this year, The New York Times famously declared 2012 the dawn of the “Age of Big Data” — an era when previously incomprehensible mountains of the world’s information can be distilled down into useful information. You’re already contributing to that mountain when you perform a Google search, buy something on Amazon, or upload a photo to Facebook. You benefit from companies refining it behind the scenes whenever Google finds exactly what you were looking for. Or a website displays an online ad for something you actually want. Or even when Facebook suggests people you already know as friends.

But the potential for Big Data goes much deeper, to the point where we may be able to calculate nearly all aspects of life.

Despite the countless papers, articles, and blog posts dedicated to the buzzword, Big Data remains a vague concept for most Web users. So for this week’s State of the Web, let’s take a look at a few of the most important aspects of this awesome, terrifying thing called Big Data, and what it could mean for the everyday person.

What is Big Data?


Big Data does not just refer to the amount of information available, but the ability of our computer systems to store and process this information economically. This evolutionary drop in the cost of computing power has taken “lots of data” and turned it into “Big Data,” which has a few important prerequisite qualities: variety, volume, and velocity. These terms were first attributed to Big Data by Gartner researcher Doug Laney in 2001 (PDF).

It’s data, in a general sense (variety): Let’s just get this first part out of the way: When people talk about Big Data, they aren’t necessarily referring to one type of information. In fact, they could be referring to any type of data: Facebook status updates, tweets on Twitter, digital images, closed-circuit camera feeds, medical histories, credit-card transactions, consumer purchasing histories, climate information, GPS location data, on and on.

If it can be stored on a computer, then it can be part of Big Data.

It’s big (volume): The key word, of course, is “big” — really big. In 2011 alone, we created or replicated 1.8 zettabytes (1.8 trillion gigabytes) of data — a number that is set to double in 2013, according to EMC (PDF). However, what constitutes “big” for one company is minuscule for another. Facebook, for example, currently stores over 100 petabytes (1 billion gigabytes) of images on its servers. The atomic physics experiments at CERN pump out 40 terabytes of data every second. But a recent study from data management company Actian Corporation shows that businesses that deal with large amounts of typically data define “big” as between 1 terabyte and 1 petabyte.

It’s quick — sometimes (velocity): A third aspect of Big Data is the rate at which information flows into a system. Twitter, for example, processes an average of about 5,000 tweets per second, according to the company’s open source manager Chris Aniszcyzyk. This number can jump significantly during high-profile events, like the Super Bowl or a major natural disaster. Other areas with high-velocity data include financial transactions, weather data, GPS coordinates, and sensor feeds from scientific equipment.

Big challenges

Big Data unstructured

The big question of Big Data is what use does all this information hold? At the moment, we don’t really know — and that’s what makes Big Data so exciting for so many industries. Companies already have mountains of information — much of it about you and me — but about 80 percent, by some estimates, is in a form that is difficult (but not impossible) for computers to “understand.” This data is called “unstructured,” and includes things like JPEG images, audio files, video files, and even many text files, including email, text messages, and blog posts. The challenge companies now face is figuring out how to turn their unstructured data into a usable information — a challenge they are quickly overcoming thanks to new applications, like Google’s BigQuery and Dremel tools.

What about now?

While we already benefit from Big Data every time we use the Web, the potential applications of Big Data extends far beyond obvious things like online search and ads. The areas where Big Data is expected to have the most immediate, revolutionary effects are business and health care.


Google and Facebook built their entire businesses on Big Data by creating services (search, connecting with friends) that are both derived from, and fueled by, massive amounts of data handed to them by users. The more features they offer, the larger their Big Data collections become, which in turn results in even more online products (to sell advertising around — a service that is itself powered by Big Data).

In other words, Google and Facebook figured out a way to monetize the data they collected. Big Data is their business. But countless other companies are looking to Big Data to provide insights into their business that were never before possible. Companies can use Big Data to tweak their advertising, prices, production operations, shipping activities, and hiring processes. At the moment, it seems, the possible uses of Big Data for business are only limited by the technical abilities of our computers and our imaginations.

Health care

In addition to making people money, Big Data is becoming increasingly useful in the field of medicine. Companies like DNAnexus and Appistry are looking to harness the vast amounts of data created by genome sequencing to help discover cures for disease far faster than has been possible.Startup Apixio is looking to bring medical records into the cloud to allow doctors to better choose treatments for their patients. Even IBM’s “Jeopardy!”-winning supercomputer Watson — which uses Big Data to power its artificial intelligence — is lending a helping hand, thanks to a partnership with WellPoint that will allow patients to access hoards of data to help them make health-care decisions.

Big and getting bigger

Big Data tsunami

The reach of Big Data doesn’t end there. Governments, scientists, militaries, and non-governmental organizations have all begun to tap into the vast power of Big Data. That power only increases as different data sets combine to offer more insight, to solve more problems, to answer deeper questions, to predict the future in ways that are impossible today. 

For average people like you and me, Big Data will provide countless new services and resources. It may even save our lives. But like all great advancements in human history, Big Data comes at a cost. Innumerable aspects of our lives — our habits, our moods, our medical histories, our personalities, our weakness and strengths, where we go, who we talk to, what we love and hate and fear — are all being amassed in nameless data centers around the world. This information may one day be used to assess whether or not you are good for a job, or a school, or whether you should have children. Companies and governments will surely know more about you and your future that you do. (In many ways, they already do.) So as Big Data gets even bigger, and the information squeezed out of it becomes more plentiful, profitable, and potent, we need to make sure this quickly moving tsunami of information doesn’t drown us in its wake.

Images via Pavel Ignatov/Carsten Reisinger/Bruce Rolff/Shutterstock


Google tracks your location — even when you deny it permission

Google is tracking your location -- even when you tell it not to. According to an investigation by the Associated Press, Google services on both Android and iPhones store location data, regardless of whether privacy settings claim…

How to transfer your contacts between iPhone and Android devices

There's nothing worse than getting a new phone and realizing you don't have any of your old contacts listed. Luckily, it's an easy problem to solve. Here's how to transfer your contact list to your new device.
Virtual Reality

Magic Leap One no longer asks for a leap of imagination. Here's how it works

The Magic Leap One AR headset is now available for a rather hefty price. Here's everything you need to know about the device including the price, where it can be purchased, the hardware powering Magic Leap's goggles and more.

The Facebook dating service will be free of charge and free of ads

Facebook is getting into the dating game. While the feature was one of the surprises from this year's F8, new details suggest what the feature may entail, including a few screenshots from a computer programmer.

Here are the best free music download sites that are totally legal

Finding music that is both free and legal to download can be difficult. We've handpicked a selection of the best free music download sites for you to legally download your next favorite album.

Google will warn businesses if state-sponsored hackers target G Suite users

Google is booting email security for G Suite subscribers. A new feature will send an alert to administrators if Google detects that a phishing or malicious email was sent to a G Suite user as a result of a government-sponsored hack.

How A.I. can defeat malware that doesn’t even exist yet

Cylance Smart Antivirus is a brand new consumer protection application that claims to only need its AI machine learning algorithm to protect you. Can ditching signatures really make for a safer future?
Emerging Tech

Automate all the little stuff in your life with these awesome IFTTT recipes

Curious about what kind of awesome things you can do with If This Then That? IFTTT recipes allow you to set up a variety of automated routines to make life easier. Check our list of the best and you'll be automating your life in no time!
Movies & TV

Tired of Netflix? Here's where to find free movies online, legally

We've spent countless hours digging around the web to find the best sites for streaming free movies online. Not only are all of these sites completely free to use, they're also completely legal and trustworthy.
Emerging Tech

Walmart’s new grocery robots aim to speed up your shopping experience

Walmart teamed up with a robot shuttle system company to find a way to speed up its in-store grocery pickup service. The service will launch in one Walmart superstore later this year.

Find your way around Google Maps with these handy tips and tricks

How good are your navigation skills? We've got a delectable menu of Google Maps tips and tricks for you right here, to take the pain out of your trips. Go from newbie to mapping master and learn how to use Google Maps.
Emerging Tech

Widespread internet access is causing mass sleep deprivation, study suggests

A study claims that high-speed internet may be costing us up to 25 minutes of sleep per night. And, surprisingly, the biggest problem isn't among those young people who are under 30.

Network routers with roaming enabled are likely susceptible to a new attack

Jens Steube discovered a new method to break into network routers while researching new ways to attack the WPA3 security standard. He stumbled onto an attack technique capable of cracking hashed WPA-PSK passwords.

Saving your favorite YouTube videos for posterity is quick, easy with these tools

Learning how to download YouTube videos is easier than you might think. There are plenty of great tools you can use, both online and offline. These are our favorites and a step by step guide on how to use them.