Data Drive Thru: Gregory Piatetsky-Shapiro

Here is an interview I gave to LatentView, discussing Analytics, Big Data, Analytics Solutions and Services, and more.

Here is an interview I gave to LatentviewLatentview:

Data Drive Thru: Gregory Piatetsky-Shapiro

"LatentView had an opportunity to Interview Gregory Piatetsky-Shapiro, Ph.D., Data Mining and Analytics Expert from KDNuggets - a one-stop target for all things related to data analytics. If you haven't heard of Gregory or have not visited KDNuggets, you are missing a treasure trove. Follow him at @kdnuggets on Twitter."

LatentView:According to you, what makes a Big Data Solutions/Services provider? Is there a minimal set of skills/capability that covers major requirements for most of the big data projects?

Gregory Piatetsky-ShapiroGregory/KDNuggets

First, I want to mention that most businesses do not have Big Data problems. Quoting a recent article by Christopher Mims for Quartz Most data isn't "big," and businesses are wasting money pretending it is .

This is also supported by a recent KDnuggets Poll, where a median answer to the question Largest Dataset Analyzed/Data Mined was in 40-60GB range, the data size which comfortably fits on a single laptop.

However, Big Data is where some of the most interesting and exciting problems are today, and it is the leading edge of technology.

Current Big Data environments require dealing not only with 3 V:

  • Big Data Volume, requiring Hadoop clusters or similar technology.
  • Big Velocity: for some problems, like Twitter analysis, military, or high-speed trading there are big and fast real-time data streams.
  • Big Variety: are there texts, images, video, links, or other inputs in addition to structured data?

But also Privacy issues, Data security, Data ownership, Data Quality, and Data governance.

However, the focus should not be on Data but on the value 'hidden' in Data. Extracting this value requires analytics and statistical skills. Also, equally important is the ability to focus on the right questions, which requires business skills. Finally, successful deployment cannot be done without organizational skills.