News

I believe that many self-proclaimed "data miners" are ignorant of statistics or at least disparage statistics because statistical knowledge and methodology represent a "black box" to them that requires mathematical training, persistence, and disciplined effort if entry is to be gained to the rich and rewarding set of tools resident within its interior. Many people are shy about numbers and mathematics in general due to bad experiences early in their education or from cultural pressures not to excel. As a result, people who can work a DM package's GUI and make it process their numbers may call themselves "data miners" but don't have the depth of statistical understanding to properly understand the fundamentals of data representation and recoding, the consequences of missing value imputation, treatment of outliers, appropriateness and limitations of algorithms, interpreting a model's results, or the host of difficulties encountered when a set of data does not conform to what their textbooks taught them.