After 150 Years, the ASA Says No to pvalues
The ASA has recently taken a position against pvalues. Read the overview and opinion of a wellrespected statistician to gain additional insight.
Sadly, the concept of pvalues and significance testing forms the very core of statistics. A number of us have been pointing out for decades that pvalues are at best underinformative and often misleading. Almost all statisticians agree on this, yet they all continue to use it and, worse, teach it. I recall a few years ago, when Frank Harrell and I suggested that R place less emphasis on pvalues in its output, there was solid pushback. One can’t blame the pusherbackers, though, as the use of pvalues is so completely entrenched that R would not be serving its users well with such a radical move.
And yet, wonder of wonders, the American Statistical Association has finally taken a position against pvalues. I never thought this would happen in my lifetime, or in anyone else’s, for that matter, but I say, Hooray for the ASA!
To illustrate the problem, consider one of the MovieLens data sets, consisting of user ratings of movies. There are 949 users. Here is an analysis in which I regress average rating per user against user age and gender:
Woohoo! Doublestar significance on age! Pvalue of only 0.004! Age is a highlysignificant predictor of movie ratings! Older people give higher ratings!
Well, no. A 10year age difference corresponds to only a 0.03 difference in ratings — quite minuscule in light of the fact that ratings take values between 1 and 5.
The problem is that with large samples, significance tests pounce on tiny, unimportant departures from the null hypothesis, in this case H0: βage = 0, and ironically declare this unimportant result “significant.” We have the opposite problem with small samples: The power of the test is low, and we will announce that there is “no significant effect” when in fact we may have too little data to know whether the effect is important.
In addition, there is the hypocrisy aspect. Almost no null hypotheses are true in the real world, so performing a significance test on them is absurd and bizarre.
Speaking of hypocrisy: As noted above, instructors of statistics courses all know of the above problems, and yet teach testing anyway, with little or (likely) no warning about this dangerous method. Those instructors also do testing in their own work.
My hat is off to ASA for finally taking some action.
Editor's note: See Norman's followup post here, which includes additional thoughts on the subject. Read the ASA's statement here.
Original. Reposted with permission.
Bio: Dr. Norm Matloff is a professor of computer science at the University of California at Davis, and was formerly a professor of statistics at that university. He is the author of The Art of R Programming (2011), Parallel Computation for Data Science (2015), and the forthcoming Linear Models to Machine Learning: Regression and Classfication, with R Examples.
Related:
 15 Mathematics MOOCs for Data Science
 Applied Statistics Is A Way Of Thinking, Not Just A Toolbox
 Amazon Top 20 Books in Statistics
Top Stories Past 30 Days

