KDnuggets Home » News » 2012 » Jun » Decline Effect in Business Analytics - failure to replicate?  (  12:n14 | Next > )

Decline Effect in Business Analytics - failure to replicate?

We can improve by learning from failures, but there are very few documented examples of business analytics failures, especially failures to replicate, where a business effect grows smaller or disappears the second time. Such examples are probably common, but people don't like to publish them. Do you know of such examples?

Are there business analytics failures - Failure to replicate the original effect?

The decline effect Recently there were stories about "The decline effect" which seemed to plague some pharmaceutical and social science studies. The effect - whatever it was - was strong in the original study, but each time the study was reproduced the effect was smaller - it "declined".

This received a lot of press in the popular media - see for example

The explanations in popular press bordered on the paranormal - "cosmic habituation", but to any experienced data scientist or data miner the most likely reason was clear - the researchers were overfitting the data in the original analysis.

See also an excellent analysis by Stanley Young from NISS Everything is Dangerous: A Controversy, June 2008, which shows many examples of bad analysis in epidemiology and medicine.

It seems that there should be similar examples in the business world, where the original effect was strong, but appeared much less strong next time when someone tried to reproduce it. One possible example is the study by Johan Bollen et al, who claim that

collective mood states derived from large-scale Twitter feeds are correlated to the value of the Dow Jones Industrial Average (DJIA) over time. ... We find an accuracy of 87.6% in predicting the daily up and down changes in the closing values of the DJIA

Althouhgh this study was soundly criticized, a hedge fund was started using that strategy.

What happened next?

That Twitter-based hedge fund? It was "quietly liquidated" a month after launch.

Please comment below or email to editor1@kdnuggets.com and I will summarize, preserving confidentiality.

Gregory Piatetsky, Editor

KDnuggets Home » News » 2012 » Jun » Decline Effect in Business Analytics - failure to replicate?  (  12:n14 | Next > )