Most Data is Not Big? A Discussion

"Most Data is not Big" post generates a very lively discussion among analytics experts - see if you agree, and join the discussion. Is collecting Big Data a distraction from focus on ROI and actionable information?

By Gregory Piatetsky-Shapiro, May 13, 2013.

This article by Christopher Mims (posted by Eric King) Most data isn't "big," and businesses are wasting money pretending it is has generated a very lively discussion on KDnuggets LinkedIn group that I wanted to share.

Dusan Toman

That's absolutely right. Besides of a few top projects (mentioned by Gregory), there is not too much really "big data" projects that are successful.

Most companies daily bread and butter is within reasonable means. They would probably do better focusing on real needs and stop talking fancy words.

I consider this big data madness a myth. If I find a spare time I'll write about it more.

Eric King

Thanks Dusan. I too plan to write more. Well... let me take a rough hack at it here in bullet form!

With Big Data, I do appreciate that the sheer volume of data generated is truly expanding with evermore momentum. And this the rate of expansion requires new methods and technologies to capture, store and retrieve all that data. I get that. It's real.

But my grievances with Big Data are largely from an analytic perspective:

  • So many organizations are simply focused on capturing anything and everything without strategy or foresight of it's potential value (or total lack thereof).
  • Many professionals in analytics believe that Big Data is all about them. Sure... Big Data is certainly preparing a greater need and platform for advanced analytics - particularly when orgs capture far more than they'll ever need to run efficiently and effectively. But at present, analytics is just one small facet of many other practices that make up the whole of Big Data. And these other practices within Big Data at the moment seem to be creating a major distraction from analytics... the true profit center. While everyone is chasing the very expensive hype of bigger buckets, bigger pipes and bigger storage tanks before this new field is even partially standardized, few seem to be looking ahead to how to extract, refine and ignite the real fuel: actionable and impactful information from the growing mass of data through data analytics.
  • If not approached properly, Bigger Data = Bigger Distortion
  • I truly hope that "Analytics 3.0" will emerge quickly and distinguish itself as the revenue-generating activity out of the massive Big Data (2.0) cost center.


I'm relieved that may not be alone in my views that the cryptic and loosely defined field of Big Data is being over-adopted and over-invested... with too little focus on organizational data use accountability and targeted analytic charters.

Those who feel that they'll never get in trouble for spending big on Big Data may be called to the carpet in a year or two. The few who spend the time to sketch out a comprehensive organizational charter for data use and data analytics - then work back into the aspects of Big Data that they truly require - will surely be elevated to higher leadership positions in the not-so-distant future.

Gregory Piatetsky-Shapiro

Eric, agree - most business and analysts don't deal with big data - this is supported by recent KDnuggets Poll where median answer to "Largest Dataset Analyzed" was 40-50 GB. However, Big Data is the leading edge of technology and research, and this is where interesting and disruptive things happen, like Google and Facebook and LinkedIn.

Dusan, you need to separate the hype around "Big Data" from tech trends that create so much "Big Data" . Big Data can improve some predictions, but will not make them perfect, because human nature has a lot of randomness. A good example is the Netflix prize, where it took 3 years to reduce the error predicting movie ranking from 0.95 stars to 0.86 stars. See my HBR blog Big Data Hype and Reality.

Most of the Big Data promise, I think, is in creating new platforms, like LinkedIn, Google, and Facebook. Also Personalized medicine, and personalized everything. Alas, Big Data also means goodbye to privacy in a digital world.