Research Library
Monthly Research
& Market Commentary

Riding the Big Data Hype Cycle

I have attended a lot of IT industry events over the years, but never one where the Gartner hype cycle was mentioned more often than at the recent Strata + Hadoop World 2012 conference in New York City. It is easy to understand why.

Big Data

While the Big Data industry is still in its early stages, its use of cloud, open source and new architectural approaches is impressive and points toward an exciting and very different future, as data volumes increase exponentially in an IT industry driven by mobility, sensors and an internet of things.

On the one hand, this was a sold-out show of some 3,000 Big Data enthusiasts keen to share their plans and experiences. The energy at the event was exceptionally high, emanating a strong sense that a new generation of business leaders is emerging. While the Big Data industry is still in its early stages, its use of cloud, open source and new architectural approaches is impressive and points toward an exciting and very different future, as data volumes increase exponentially in an IT industry driven by mobility, sensors and an internet of things.

But as most of the speakers and attendees were not twenty-somethings, they were also wary of grandiosity.

People visibly winced at the easy talk about a new “age of understanding” and, of course, “a management revolution”. As in the early days of the web, there were many predictions about diseases being cured, education being re-invented, governments becoming more transparent, and social cohesion improving. Big Data’s appearance on the cover of the October Harvard Business Review is another sign that we may have reached the peak of the hype cycle.

We think a good way to reconcile these two perspectives is by distinguishing between information and decision-making. The former is improving rapidly, the latter much more slowly. Perhaps the most succinct example of this came from Samantha Ravich, who has spent many years in the upper levels of the US intelligence community. At Strata, she used an example from Afghanistan, where the US has highly detailed maps and other information about the Afghan poppy fields, but finds this data to be of no help whatsoever in deciding what to do. Burn the fields? Buy the poppies? Pay people to grow food? Do nothing? These questions still come down to human judgments and intuition, even guesswork.

While Afghanistan might be viewed as an extreme case, we think it is instructive. We strongly agree that businesses will become more data-driven and outside-in, with ever-better mobile and smart sensing mechanisms that help us see what is really going on. But, as with knowledge management and data warehousing in the past, we have doubts about the scale of both the near- and long-term impact. Below are just a few of the apparent limitations:

  • Current case examples aren’t that compelling. At Strata, most of the sessions were about new tools and technologies, mostly aimed at making the Hadoop platform more complete and easy to use. But I made a particular effort to attend those sessions featuring customers – in industries as diverse as insurance, real estate, finance, telecommunications, health care, government and entertainment. While all the customer speakers touted real benefits and were enthusiastic about the future, the overall business impact thus far has been modest. We have observed a similar pattern in our LEF research. There has been little of the traditional low-hanging fruit.
  • It is often easy to catch up. The book and movie Moneyball is the most widely cited example of Big Data in action, especially in US government circles. Computers and statistics were used to successfully challenge and refute many traditional baseball beliefs and practices. But today, just about every baseball team uses sabermetrics, and the advantages of the newer, more sophisticated metrics are diminishing. Baseball management strategies are shifting more toward human factors such as player development and the risks/ incentives that come with long-term player contracts. Will most Big Data projects also produce easily copied gains?
  • Customer inertia is much higher than the IT industry is used to. One of the great hopes of the Big Data movement is that we will be able to use advanced technology for things more important than financial trading and serving online ads. But while the potential in areas such as health care, education and government is great, so is the inertia, and many of the speakers at Strata were frank about the challenges in significantly changing the behaviour of doctors, professors, judges, police departments, labour unions, government officials and other entrenched interests. Without market forces, the incentive to change can be insufficient, although rising cost pressures in all of these areas may help.
  • The talent shortage is real. Speakers were generally optimistic that today’s data scientist shortage would eventually be addressed through a combination of more user-friendly tools, more targeted university and online training, and improved company practices for assembling Big Data teams.
    Organizations are starting to realize that, while they might not have a Big Data guru, they can put together small teams with the necessary combination of business, statistical, research and technology skills. Nevertheless, firms still have a long way to go, and thus they tend to rely heavily on suppliers – be they giants such as IBM and EMC or specialists such as Cloudera, Platfora and many others. At Strata, many of the customer speakers almost had their hands held by their vendor sponsors.

What should business/IT leaders make of all of this?

Given all of this, how should we see and discuss the impact of Big Data? We like to start by choosing an appropriate metaphor that captures the current situation. Our three main candidates thus far are summarized below:

  1. There’s gold in them there hills! This is the most common metaphor, and is implicit in the term data mining. Big Data proponents (especially storage vendors) argue that there is great value trapped in our mountains of corporate and web-based information, and like to cite examples of significant discoveries.
    But naysayers have also adopted the gold rush metaphor, pointing out that in California in the 1850s most of the discoveries turned out to be fool’s gold (pyrite), and that most of the profits were made by those selling boots, pick axes, denim jeans and whiskey. Were these the predecessors of today’s Hadoop clusters, in-memory systems, business intelligence software and services, and of course, coffee?
  2. Information as fuel. Our newest research associate, Venkatesh Rao, has been developing an interesting analogy with the energy industry. For more than a century, oil companies released or burned off natural gas in order to get safely to the nearby oil. They did this (and sometimes still do) because they didn’t have the technology to capture, distribute and use the gas, and thus they had to throw away a resource that would eventually prove to be nearly as valuable as the oil itself. In this view, we have long been releasing vast quantities of digital exhaust, which today’s Big Data industry seeks to harness effectively to create new industries and power the economy of the future.
  3. Information hygiene. In business today, information is like the air we breathe – essential, pervasive and mostly invisible. In this metaphor, it is the quality of the air that matters. Does the information that an organization ‘breathes’ help to keep it alert and vigorous, or does poor/dirty information tend to diminish our senses and sap our vitality? This suggests that the effective use of Big Data is less about a management revolution and more a way of staying more fit and aware than one’s competitors.

Which of these metaphors, if any, works best for you? Do you see others? While there is truth in all of them, based on what I saw at Strata, I tend to lean toward the information hygiene analogy, as it is less binary than the gold/pyrite and waste/resource perspectives. It is also more incremental (and so less revolutionary) in nature. Businesses have always been data-driven in that they must respond to market signals; the Big Data movement is about making these signals stronger and more accurate. Most of the examples I heard at Strata were along the hygiene lines. The more radical changes tended to be based on open data, which is sometimes Big and sometimes not.

Limits to knowledge

But no matter which of these perspectives you prefer, keep in mind that there will always be limits. As Thomas Edison observed: “It’s obvious that we don’t know one millionth of one percent about anything.” While perhaps we know a bit more than this now, the fabled “single source of truth” is anything but right around the corner. Thus far, there is little evidence that more information results in a stronger group or social consensus. It often just raises more and better questions.

While traditional economic definitions and methodologies are clearly outdated, modern outside-in Big Data approaches are not yet close to taking their place.

Consider that despite armies of economists and statisticians with unlimited access to modern IT, even our most fundamental economic measurements – GDP, growth, unemployment, inflation and productivity – are not sufficiently timely or reliable; let alone our inability to forecast or act on such things. While traditional economic definitions and methodologies are clearly outdated, modern outside-in Big Data approaches are not yet close to taking their place.

Most businesses find themselves in a similar situation. They know that many of their traditional metrics are flawed, but they also know that the data-driven, real-time enterprise of the future is still much more promise than reality. To use our hygiene analogy, today’s Big Data mandate is like the passage of the Clean Air Acts in the UK in 1956 and the US in 1963. It’s a real sign of progress and a major turning point, even though the ultimate goal will probably never be fully achieved.


*{{ error }}
*{{ error }}
*{{ error }}
*{{ error }}
*{{ error }}
*{{ error }}


Research Commentary

PDF (801.3 KB)


David Moschella
Research Fellow
David Moschella, based in the United States, is a Research Fellow for Leading Edge Forum.  David's focus is on industry disruptions, machine intelligence and related business model strategies.  David was previously Global Research Director of the programme. David’s key areas of expertise include globalization, industry restructuring, disruptive technologies, and the co-evolution of business and IT.  He is the author of multiple research reports, including Disrupting the Professions through Machine Learning and Digital Trust, 2016 Study Tour Report: Applying Machine Intelligence, There is Now a Formula for Machine Intelligence Innovation,  Embracing 'the Matrix' and the Machine Intelligence Era and The Myths and Realities of Digital Disruption. An author and columnist, David’s second book, Customer-Driven IT, How Users Are Shaping Technology Industry Growth, was published in 2003 by Harvard Business School Press.  The book predicted the shift from a supplier-driven to today’s customer-led IT environment.  His 1997 book, Waves of Power, assessed global competition within the IT supplier community.  He has written some 200 columns for Computerworld, the IT Industry’s leading publication on Enterprise IT, and has presented at countless industry events all around the world. David previously spent 15 years with International Data Corporation, where he was IDC’s main spokesperson on global IT industry trends and was responsible for its worldwide technology, industry and market forecasts.    


21st Century
Adaptive Execution
Proactive, Haptic Sensing
Reimagining the Portfolio
Value Centric Leadership


IoT is Turning the World into a Soft Machine
25 Aug 2017 / By Dave Aron
So Much is Becoming Possible with ‘Matrix’ Technologies – How Will Your Organization Take Advantage?
23 Aug 2017 / By Glen Robinson
A Summer Contemplation: Digital Extensions and Platform Businesses
03 Aug 2017 / By Dave Aron
Jaws 2018 – The Digerati Bite the Banks
17 Jul 2017 / By Bill Murray
“Hey, IT Department – What Do You Want to Be When You Grow Up?”
06 Jul 2017 / By Dave Aron