Go to Top

The Perfect Storm: The Impact of Analytics, Big Data and Cloud

In the early 1960s, Edward Lorenz, the American meteorologist and pioneer of chaos theory suggested that the flapping of a butterfly’s wings in some far-flung jungle could trigger a hurricane on the other side of the world. Lorenz was making two points, the second of which is often missed. The first mathematically observed fact was that very small perturbations in initial parameters can have unexpectedly enormous consequences in the outcome of a model. The second point was that tracing the consequential chain from cause to effect as the basis of forecasting is well nigh impossible. Which jungle should we visit; which butterfly should we observe; which eddies of air rising from its flapping wings should we trace? In fact, even using the concepts of cause and effect in this instance may seriously misrepresent reality.

Watch The Briefing Room with Barry Devlin and NuoDB.

In the world of business intelligence – or what used to be called business intelligence – we are indeed living in the heart of a perfect storm, a butterfly-induced event so significant that it may even be climate change … and the creation of a comprehensive biz-tech ecosystem, where technology advances and business advantage are symbiotically linked, such that neither one can exist or thrive without the other.

 

 

A trio of tiny technological butterflies flutters in the origins of this storm. The miniaturization and subsequent commoditization of computing, sensor and communications devices have together created a computing environment described by the suitably nebulous term, the cloud. We are in the process of moving from a paradigm in which IT knew where each data file was located, on which CPU it would be processed and by which instantiation of whose code to one where all these certainties disappear.

Virtualized everything. Of course, it’s a trend that has been emerging for many years. The concept – and implementation – of virtual machines dates back at least to the 1960s and CP/CMS (later VM/CMS) by IBM. Elastic computing, in some senses, is a modern incarnation of time-sharing. The difference is that the environment has become so complex that no single point of control exists any longer; no in-house IT department can manage the entire computing scope of the business. While some analysts predict the demise of in-house IT as a consequence of the cloud, I disagree. However, IT must change its vision and role. From being the setter of technology strategy and provider of hardware and software tools, IT must become a consultant to the business on how technology advances can enable business innovation and the knowledgeable arbitrator between technology providers and business purchasers: partners in the biz-tech ecosystem.

These same three butterflies also flap their wings in the genesis of a new information environment. The phrase big data is already hackneyed, but totally inadequate to describe the emerging situation. The number of devices alone, nevermind the amount of data each generates, is staggering: some 10 billion devices connected to the Internet today, rising to 25 billion by 2020, according to the current ITU annual report. Already in 2012, there are some 30 billion RFID tags embedded in our products, farm animals and even our bodies. Today, we can gather, store and process information at levels of detail, speed and size previously unimaginable.

Much of the focus so far on big data has been around storage and processing. The widely touted three Vs – volume, variety and velocity – express technology constraints that must be addressed: do we have enough storage, processing power and appropriate software algorithms to work with this flood of information? But this question begs another, more fundamental one. What should we do with the information? Business, long accustomed to the “here’s my problem/idea; how can I get the data?” approach, must now have the vision to ask “here’s the data; what can I do with it?” and partner with IT to understand and evaluate the possibilities: the biz-tech ecosystem again.

So we come to analytics – business analytics, operational analytics, predictive analytics – whatever-you’re-having-yourself analytics. And, if we are to believe the current hype, it’s about to reinvent business. Those who analyze win. In 2011, Erik Brynjolfsson, an economist at the Massachusetts Institute of Technology (MIT), and colleagues concluded that firms that make decisions based on data and analytics performed 5 to 6% better than those that rely on intuition and experience. This data-driven study on “data-driven decision making” at 179 large publicly traded firms is often quoted by vendors and analysts to confirm the value of data collection and analysis. As a general principle, of course it does. Unfortunately, in my view, it fails to address the question of which data to collect and what analysis to perform. These are fundamental questions to ask in a world where information is more than a commodity, where data is omnipresent and its sources, quality and meaning left largely to chance.

The phrase perfect storm is often a positive one in the IT industry – the biz-tech ecosystem is my upbeat use. However, its underlying meaning is rather more grim: an event where a rare combination of circumstances will aggravate a situation drastically.” Which should we apply here?

To finally return to Lorenz’s butterfly analogy, consider the problem it poses about cause and effect. A new model of analytics-driven thinking is being proposed that might be summarized as “let the data speak for itself.” The suggestion is that we have (or soon will have) enough data and enough processing power that we can simply allow statistically valid patterns to emerge from the data and automatically (at least in some cases) allow the computers to make the decisions. In fact, we see this already in automated stock trading, and even within this very limited domain there have been some unexplained anomalies. The underlying thinking is that we can find that butterfly and predict the future based on analyzing how it flaps it wings. That, I predict, is a butterfly that may defy nature and turn into an extremely toxic caterpillar.

About Barry Devlin

Dr. Barry Devlin, Founder and Principal of 9sight Consulting, provides strategic consulting and thought leadership to buyers and vendors of BI solutions. He is a founder of the data warehousing industry and among the foremost authorities worldwide on business intelligence (BI) and beyond. Barry is a widely respected consultant, lecturer and author of “Data Warehouse — from Architecture to Implementation.”

About Barry Devlin

Dr. Barry Devlin, Founder and Principal of 9sight Consulting, provides strategic consulting and thought leadership to buyers and vendors of BI solutions. He is a founder of the data warehousing industry and among the foremost authorities worldwide on business intelligence (BI) and beyond. Barry is a widely respected consultant, lecturer and author of “Data Warehouse — from Architecture to Implementation.”

One Response to "The Perfect Storm: The Impact of Analytics, Big Data and Cloud"

  • Girish Gurudutt
    November 9, 2012 - 9:40 am Reply

    Nice post; we are in the era where information is available and can be used to create value. we have to make use of information and create value in the shortest time possible.

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>