Nine Laws of Data Mining, part 2

8th Law of Data Mining – “Value Law”:

 The value of data mining results is not determined by the accuracy or stability of predictive models

Accuracy and stability are useful measures of how well a predictive model makes its predictions.  Accuracy means how often the predictions are correct (where they are truly predictions) and stability means how much (or rather how little) the predictions would change if the data used to create the model were a different sample from the same population.  Given the central role of the concept of prediction in data mining, the accuracy and stability of a predictive model might be expected to determine its value, but this is not the case.   

The value of a predictive model arises in two ways:

• The model’s predictions drive improved (more effective) action, and
• The model delivers insight (new knowledge) which leads to improved strategy.

In the case of insight, accuracy is connected only loosely to the value of any new knowledge delivered.  Some predictive capability may be necessary to convince us that the discovered patterns are real.  However, a model which is incomprehensibly complex or totally opaque may be highly accurate in its predictions, yet deliver no useful insight, whereas a simpler and less accurate model may be much more useful for delivering insight.

The disconnect between accuracy and value in the case of improved action is less obvious, but still present, and can be highlighted by the question “Is the model predicting the right thing, and for the right reasons?”  In other words, the value of a model derives as much from of its fit to the business problem as it does from its predictive accuracy.  For example, a customer attrition model might make highly accurate predictions, yet make its predictions too late for the business to act on them effectively.  Alternatively an accurate customer attrition model might drive effective action to retain customers, but only for the least profitable subset of customers.  A high degree of accuracy does not enhance the value of these models when they have a poor fit to the business problem.

The same is true of model stability; although an interesting measure for predictive models, stability cannot be substituted for the ability of a model to provide business insight, or for its fit to the business problem.  Neither can any other technical measure.

In summary, the value of a predictive model is not determined by any technical measure.  Data miners should not focus on predictive accuracy, model stability, or any other technical metric for predictive models at the expense of business insight and business fit.

9th Law of Data Mining – “Law of Change”:

 All patterns are subject to change

The patterns discovered by data mining do not last forever.  This is well-known in many applications of data mining, but the universality of this property and the reasons for it are less widely appreciated.

In marketing and CRM applications of data mining, it is well-understood that patterns of customer behaviour are subject to change over time. Fashions change, markets and competition change, and the economy changes as a whole; for all these reasons, predictive models become out-of-date and should be refreshed regularly or when they cease to predict accurately.

The same is true in risk and fraud-related applications of data mining.  Patterns of fraud change with a changing environment and because criminals change their behaviour in order to stay ahead of crime prevention efforts.  Fraud detection applications must therefore be designed to detect new, unknown types of fraud, just as they must deal with old and familiar ones.

Some kinds of data mining might be thought to find patterns which will not change over time – for example in scientific applications of data mining, do we not discover unchanging universal laws?  Perhaps surprisingly, the answer is that even these patterns should be expected to change.

The reason is that patterns are not simply regularities which exist in the world and are reflected in the data – these regularities may indeed be static in some domains.  Rather, the patterns discovered by data mining are part of a perceptual process, an active process in which data mining mediates between the world as described by the data and the understanding of the observer or business expert.  Because our understanding continually develops and grows, so we should expect the patterns also to change.  Tomorrow’s data may look superficially similar, but it will have been collected by different means, for (perhaps subtly) different purposes, and have different semantics; the analysis process, because it is driven by business knowledge, will change as that knowledge changes.  For all these reasons, the patterns will be different.

To express this briefly, all patterns are subject to change because they reflect not only a changing world but also our changing understanding.

Postscript

The 9 Laws of Data Mining are simple truths about data mining. Most of the 9 laws are already well-known to data miners, although some are expressed in an unfamiliar way (for example, the 5th, 6th and 7th laws). Most of the new ideas associated with the 9 laws are in the explanations, which express an attempt to understand the reasons behind the well-known form of the data mining process.

Why should we care why the data mining process takes the form that it does? In addition to the simple appeal of knowledge and understanding, there is a practical reason to pursue these questions.

The data mining process came into being in the form that exists today because of technological developments – the widespread availability of machine learning algorithms, and the development of workbenches which integrated these algorithms with other techniques and make them accessible to users with a business-oriented outlook. Should we expect technological change to change the data mining process? Eventually it must, but if we understand the reasons for the form of the process, then we can distinguish between technology which might change it and technology which cannot.

Several technological developments have been hailed as revolutions in predictive analytics, for example the advent of automated data preparation and model re-building, and the integration of business rules with predictive models in deployment frameworks. The 9 laws of data mining suggest, and their explanations demonstrate, that these developments will not change the nature of the process. The 9 laws, and further development of these ideas, should be used to judge any future claims of revolutionising the data mining process, in addition to their educational value for data miners.

I would like to thank Chris Thornton and David Watkins, who supplied the insights which inspired this work, and also to thank all those who have contributed to the LinkedIn “9 Laws of Data Mining” discussion group, which has provided invaluable food for thought.

Bio:
Tom Khabaza helps organisations improve their marketing and customer processes, to improve their efficiency, risk analysis and fraud detection, and to improve their strategic decision-making, through new knowledge and predictive capabilities extracted from data. Tom has worked in the field of data mining for over 20 years, and is one of the authors of the world-leading Clementine data mining workbench, and of the CRISP-DM industry standard data mining methodology.

Original. Reposted by permission.

Related

• Nine Laws of Data Mining, part 1
• CRISP-DM, still the top methodology for analytics, data mining, or data science projects
• Active Data Mining, Data Science blogs