We can improve by learning from failures, but there are very few documented examples of business analytics failures, especially failures to replicate, where a business effect grows smaller or disappears the second time. Such examples are probably common, but people don't like to publish them. Do you know of such examples?
Are there business analytics failures - Failure to replicate the original effect?
Recently there were stories about
"The decline effect"
which seemed to plague some pharmaceutical
and social science studies. The effect - whatever it was - was strong in the original study, but each time the study was reproduced the effect was smaller - it "declined".
This received a lot of press in the popular media - see for example
The explanations in popular press bordered on the paranormal - "cosmic habituation", but to any experienced data scientist or data miner
the most likely reason was clear - the researchers were overfitting the data in the original analysis.
See also an excellent analysis by Stanley Young from NISS
Everything is Dangerous: A Controversy, June 2008, which shows many examples of bad analysis in epidemiology and medicine.
It seems that there should be similar examples in the business world, where
the original effect was strong, but appeared much less strong next time when someone tried to reproduce it.
One possible example is the
study by Johan Bollen et al, who claim that
collective mood states derived from large-scale Twitter feeds are correlated to the value of the Dow Jones Industrial Average (DJIA) over time. ... We find an accuracy of 87.6% in predicting the daily up and down changes in the closing values of the DJIA
Althouhgh this study was
soundly criticized, a hedge fund was started using that strategy.
What happened next?
That Twitter-based hedge fund? It was "quietly liquidated" a month after launch.
Please comment below or email to firstname.lastname@example.org and I will summarize, preserving confidentiality.
Gregory Piatetsky, Editor