Stats Can Trip Up Scientific Data Fabricators

... The psychology research community is already reeling from a report that Dirk Smeesters, PhD, a professor of consumer behavior and society at Erasmus University in Rotterdam, the Netherlands, manipulated data in his 2011 paper, "The effect of color (red versus blue) on assimilation versus contrast in prime-to-behavior effects" in the Journal of Experimental Psychology.

A statistical method developed by Uri Simonsohn, PhD, identified patterns in the raw data underlying work by Smeesters that strongly suggested they were not real, according to a report from Committee for Inquiry into Scientific Integrity at Erasmus University.

Simonsohn, who is at the Wharton School of the University of Pennsylvania in Philadelphia, told university officials of his suspicions, which conducted its own investigation and determined that something was indeed amiss. They concluded that the data were statistically highly unlikely.

Smeesters denied having made up the suspect data, but was unable to provide the raw data behind the findings, asserting the files were lost in a computer crash after he had sent a copies to Simonsohn.

But he did admit to selective omission of data points that undercut the hypotheses he was promoting. However, he insisted that such omission was common practice in psychology and marketing research.

Read more.

See also Fraud Detection Method Called Credible But Used Like an 'Instrument of Medieval Torture'

Simonsohn's method has not been published, but here is a summary from Richard Gill of Leiden University, who also provides R-code for Simonsohn's fraud test

Simonsohn's idea [according to the report by the investigation commission] is that if extreme data has been removed in an attempt to decrease variance and hence boost significance, the variance of sample averages will decrease. Now researchers in social psychology typically report averages, sample variances, and sample sizes of subgroups of their respondents, where the groups are defined partly by an intervention (treatment/control) and partly by covariates (age, sex, education ...). So if some of the covariates can be assumed to have no effect at all, we effectively have replications: i.e., we see group averages, sample variances, and sample sizes, of a number of groups whose true means can be assumed to be equal. Simonsohn's test statistic for testing the null-hypothesis of honesty versus the alternative of dishonesty is the sample variance of the reported averages of groups whose mean can be assumed to be equal. The null distribution of this statistic is estimated by a simulation experiment, by which I suppose is meant a parametric bootstrap.