The New Big Data

Top scientists from companies such as Google and Yahoo are gathered alongside leading academics at the 17th Association for Computing Machinery (ACM) conference on Knowledge Discovery and Data Mining (KDD) in San Diego this week. They will present the latest techniques for wresting insights from the deluge of data produced nowadays, and for making sense of information that comes in a wider variety of forms than ever before.

Twenty years ago, the only people who cared about so-called "big data"-the only ones who had enormous data sets and the motivation to try to process them-were members of the scientific community, says Usama Fayyad, executive chair of ACM's Special Interest Group on Knowledge Discovery and Data Mining and former chief data officer at Yahoo. Even then, the results of data mining were impressive. "We were able to solve significant scientific problems that were standing in the field for 30-plus years," Fayyad says.

"Businesses and industry are increasingly interested in leveraging the data they capture through business processes," says Chid Apte, director of analytics research at IBM and chair of the conference. In particular, he points to health care, social media, and anything that takes place on the Web.

These days, Internet giants make their money from the information they collect about users and the insights they gain from mining it. Retailers can access complex patterns of shopper behavior to help them stock their stores more profitably. Industry researchers can predict automobile traffic patterns based on congestion, weather, and time of year, and offer the best routes.

Read more.