The Data Revolution

If you haven't read McKinsey Global Institute's report on Big Data (or at least the executive summary), let me try to convey how important a role data will play in the coming years. Every application has to make use of data, and many industries have built up a lot of data over the last few decades. The amount of data stored is expected to continue to grow at an astonishing 50% per year as storage space becomes cheaper and as our ability to access it becomes easier with cloud technologies. Growth rates are predicted to be 20% for structured data (databases) and 60% for unstructured data (documents, messages, pictures, videos). Our ability to store, architect, and analyze data will impact every major industry, not just Wall Street. Consider a few examples.

TaKaDu, an Israeli tech firm has used data that already exists to better predict leaks in water mains, as well as diversions of the water supply. No new sensors are needed--just better analysis of the data at hand. As water struggles become more intense over the coming years, more efficient use of water is key.

The legal world is being changed through artificial intelligence and data mining as well. Blackstone specializes in e-discovery and provides a cheap alternative to expensive law firms and their armies of 1st and 2nd year lawyers. Why pay people to look at thousands of documents when computers can scan millions?

One of the reasons Tesco has dominated the British groceries market is its ability to understand its customers. It does this through collecting information about purchases and then providing price adjustments and customized discounts. Individual stores are tailored according to the demographics of their customers.

Another interesting example involves policing. As funding is cut, police need to be more intelligent about where they patrol. Smart Policing helps police analyze trends to predict where crime will occur in the future, so less time is needed on the beat.

The data revolution will impact health-care, government, marketing, education--you name it. And, of course, all businesses are working internally to deal with the data deluge. Coworkers need need to be able to share documents in a way that is productive. Products like SharePoint and MarkLogic have become increasingly popular to deal with such unstructured data.

The exponential increase in data and the need to wrangle it will be so great that we will struggle to cope for years. MGI predicts that by 2018 there will be a shortage of 140,000 data professionals in the US alone. It's not just DBAs and architects, but philosophers, analysts, miners, mavens, and entrepreneurs that are needed. One reason that Google and Facebook have become so popular is that they help filter out the noise. I don't pretend to know where this is all going, but I can say I'm excited to be a part of the revolution.