Breaking down ‘Big Data’ and Internet in the age of variety, volume, and velocity

State of the Web: Boiling down Big Data

Earlier this year, The New York Times famously declared 2012 the dawn of the “Age of Big Data” — an era when previously incomprehensible mountains of the world’s information can be distilled down into useful information. You’re already contributing to that mountain when you perform a Google search, buy something on Amazon, or upload a photo to Facebook. You benefit from companies refining it behind the scenes whenever Google finds exactly what you were looking for. Or a website displays an online ad for something you actually want. Or even when Facebook suggests people you already know as friends.

But the potential for Big Data goes much deeper, to the point where we may be able to calculate nearly all aspects of life.

Despite the countless papers, articles, and blog posts dedicated to the buzzword, Big Data remains a vague concept for most Web users. So for this week’s State of the Web, let’s take a look at a few of the most important aspects of this awesome, terrifying thing called Big Data, and what it could mean for the everyday person.

What is Big Data?


Big Data does not just refer to the amount of information available, but the ability of our computer systems to store and process this information economically. This evolutionary drop in the cost of computing power has taken “lots of data” and turned it into “Big Data,” which has a few important prerequisite qualities: variety, volume, and velocity. These terms were first attributed to Big Data by Gartner researcher Doug Laney in 2001 (PDF).

It’s data, in a general sense (variety): Let’s just get this first part out of the way: When people talk about Big Data, they aren’t necessarily referring to one type of information. In fact, they could be referring to any type of data: Facebook status updates, tweets on Twitter, digital images, closed-circuit camera feeds, medical histories, credit-card transactions, consumer purchasing histories, climate information, GPS location data, on and on.

If it can be stored on a computer, then it can be part of Big Data.

It’s big (volume): The key word, of course, is “big” — really big. In 2011 alone, we created or replicated 1.8 zettabytes (1.8 trillion gigabytes) of data — a number that is set to double in 2013, according to EMC (PDF). However, what constitutes “big” for one company is minuscule for another. Facebook, for example, currently stores over 100 petabytes (1 billion gigabytes) of images on its servers. The atomic physics experiments at CERN pump out 40 terabytes of data every second. But a recent study from data management company Actian Corporation shows that businesses that deal with large amounts of typically data define “big” as between 1 terabyte and 1 petabyte.

It’s quick — sometimes (velocity): A third aspect of Big Data is the rate at which information flows into a system. Twitter, for example, processes an average of about 5,000 tweets per second, according to the company’s open source manager Chris Aniszcyzyk. This number can jump significantly during high-profile events, like the Super Bowl or a major natural disaster. Other areas with high-velocity data include financial transactions, weather data, GPS coordinates, and sensor feeds from scientific equipment.

Big challenges

Big Data unstructured

The big question of Big Data is what use does all this information hold? At the moment, we don’t really know — and that’s what makes Big Data so exciting for so many industries. Companies already have mountains of information — much of it about you and me — but about 80 percent, by some estimates, is in a form that is difficult (but not impossible) for computers to “understand.” This data is called “unstructured,” and includes things like JPEG images, audio files, video files, and even many text files, including email, text messages, and blog posts. The challenge companies now face is figuring out how to turn their unstructured data into a usable information — a challenge they are quickly overcoming thanks to new applications, like Google’s BigQuery and Dremel tools.

What about now?

While we already benefit from Big Data every time we use the Web, the potential applications of Big Data extends far beyond obvious things like online search and ads. The areas where Big Data is expected to have the most immediate, revolutionary effects are business and health care.


Google and Facebook built their entire businesses on Big Data by creating services (search, connecting with friends) that are both derived from, and fueled by, massive amounts of data handed to them by users. The more features they offer, the larger their Big Data collections become, which in turn results in even more online products (to sell advertising around — a service that is itself powered by Big Data).

In other words, Google and Facebook figured out a way to monetize the data they collected. Big Data is their business. But countless other companies are looking to Big Data to provide insights into their business that were never before possible. Companies can use Big Data to tweak their advertising, prices, production operations, shipping activities, and hiring processes. At the moment, it seems, the possible uses of Big Data for business are only limited by the technical abilities of our computers and our imaginations.

Health care

In addition to making people money, Big Data is becoming increasingly useful in the field of medicine. Companies like DNAnexus and Appistry are looking to harness the vast amounts of data created by genome sequencing to help discover cures for disease far faster than has been possible.Startup Apixio is looking to bring medical records into the cloud to allow doctors to better choose treatments for their patients. Even IBM’s “Jeopardy!”-winning supercomputer Watson — which uses Big Data to power its artificial intelligence — is lending a helping hand, thanks to a partnership with WellPoint that will allow patients to access hoards of data to help them make health-care decisions.

Big and getting bigger

Big Data tsunami

The reach of Big Data doesn’t end there. Governments, scientists, militaries, and non-governmental organizations have all begun to tap into the vast power of Big Data. That power only increases as different data sets combine to offer more insight, to solve more problems, to answer deeper questions, to predict the future in ways that are impossible today. 

For average people like you and me, Big Data will provide countless new services and resources. It may even save our lives. But like all great advancements in human history, Big Data comes at a cost. Innumerable aspects of our lives — our habits, our moods, our medical histories, our personalities, our weakness and strengths, where we go, who we talk to, what we love and hate and fear — are all being amassed in nameless data centers around the world. This information may one day be used to assess whether or not you are good for a job, or a school, or whether you should have children. Companies and governments will surely know more about you and your future that you do. (In many ways, they already do.) So as Big Data gets even bigger, and the information squeezed out of it becomes more plentiful, profitable, and potent, we need to make sure this quickly moving tsunami of information doesn’t drown us in its wake.

Images via Pavel Ignatov/Carsten Reisinger/Bruce Rolff/Shutterstock


5G is the swift kick VR and AR gaming needs to come to fruition

There's a lot of hype surrounding augmented reality and virtual reality, but is it really the next big thing? We take a look at where the new mediums stand, as well as how 5G is poised to help them break into the mainstream.
Smart Home

OK Google, what else can you do? The best tips and tricks for Google Home

The Home functions in a similar fashion to its main competitor, the Amazon Echo, but has the added benefit of select Google services. Here are few tips to help you make the most of the newfangled device.
Emerging Tech

Here’s how Facebook taught its Portal A.I. to think like a Hollywood filmmaker

When Facebook introduced its Portal screen-enhanced smart speakers, it wanted to find a way to make video chat as intimate as sitting down for a conversation with a friend. Here's how it did it.
Smart Home

Google Home and Amazon Alexa are asking smart home device makers for user info

Google and Amazon want to establish a "continuous flow" of information between their servers and your smart home devices, but companies like Logitech have begun to speak out for user privacy.

Marriott asking guests for data to see if they were victims of the Starwood hack

Marriott has created an online form to help you find out if your data was stolen in the massive Starwood hack that came to light toward the end of 2018. But take note, it requires you to submit a bunch of personal details.

New Chrome feature aimed at preventing websites from blocking Incognito Mode

A new Chrome feature will prevent websites from blocking Chrome users as they browse using Incognito Mode. The feature is supposed to fix a known loophole that allows websites to detect and block those using Incognito Mode.

Microsoft extension adds Google Chrome support for Windows Timeline

The Windows Timeline feature is now much more versatile thanks to the added support for Google's Chrome browser. All you need to do to increase its functionality is to download the official Chrome extension.

Reluctant to give your email address away? Here's how to make a disposable one

Want to sign up for a service without the risk of flooding your inbox with copious amounts of spam and unwanted email? You might want to consider using disposable email addresses via one of these handy services.

Chrome is a fantastic browser, but is is still the best among new competitors?

Choosing a web browser for surfing the web can be tough with all the great options available. Here we pit the latest versions of Chrome, Opera, Firefox, Edge, and Vivaldi against one another to find the best browsers for most users.
Movies & TV

Here’s how to watch the 2019 Oscars livestream online

The 91st Academy Awards will air live on ABC, but there are also a number of ways to watch Hollywood's biggest night online using your mobile device, desktop, or set-top streamer. Here's how to catch the Oscars livestream.

YouTube changes its strikes system, offers softer first-offense penalty

YouTube announced changes to its strikes system for its content creators. The changes include a softer first-offense penalty for creators who violate YouTube's guidelines and more consistent penalties for further violations.

An experimental feature could help reduce memory usage in Google Chrome

Google Chrome might be the most popular web browser, but it also is a resource hog. Google is currently working on an experimental feature for Chrome which sets out to reduce its overall memory usage. 

Need a free alternative to Adobe Illustrator? Here are our favorites

Photoshop and other commercial tools can be expensive, but drawing software doesn't need to be. The best free drawing software is just as powerful as some of the more expensive offerings.

Edit, sign, append, and save with six of the best PDF editors

Though there are plenty of PDF editors to be had online, finding a solution with the tools you need can be tough. Here are the best PDF editors for your editing needs, no matter your budget or OS.