Data hoarding and bias among big challenges in big data and analytics

Hype aside, exploiting big data and analytics will matter hugely to companies' future performance, remaking whole industries and spawning new ones. The list of challenges is long, however. They range from the well-documented paucity of data scientists available to crunch that big data, to more intractable but less-mentioned problems rooted in human nature.

One of the latter is humans' tendency to hoard data. Another is their tendency to hold on to preconceived beliefs even when the data screams otherwise. That was the consensus of a panel of data experts speaking on big data and analytics at the recent MIT Sloan CIO Symposium in Cambridge, Mass.

Another landmine? False hope. There is no final truth in big data and analytics, as the enterprises that do big data well already know. Iteration is all, the panel agreed.

Moreover, except for the value of iteration, CIOs can forget about best practices. Emerging so-called next practices are about the best companies can lean on as they dive into big data, said computer scientist Michael Chui, San Francisco-based senior fellow at the McKinsey Global Institute, the research arm of New York-based McKinsey & Co. Inc.

"The one thing we know that doesn't work: Wait five years until the perfect data warehouse is ready," said Chui, who's an author of last year's massive McKinsey report on the value of big data.

Seeing data quality in relative terms
In fact, obsessing over data quality is one of the first hurdles many companies have to overcome if they hope to use big data effectively, Chui said. Data accuracy is of paramount importance in banks' financial statements. Messy data, however, contains patterns that can highlight business problems or provide insights that generate significant value, as laid out in a related story about the symposium panel, "Seize big data and analytics or fall behind, MIT panel says."

Read more.