Search Results

This chapter reviews and evaluates reports that scientists often cannot repeat, or “reproduce” published work. It begins by defining what “reproducibility” means and how reproducibility applies to various kinds of science. The focus then shifts to the Reproducibility Project: Psychology, which was a systematic effort to repeat published findings in psychology, and which gave rise to many of the present concerns about reproducibility. The chapter critically examines the Reproducibility Project and points out how the nature of science and the complexity of nature can stymie the best attempts at reproducibility. The chapter also reviews the statistical criticisms of science that John Ioannidis and Katherine Button and their colleagues have raised. The hypothesis is a central issue because it is inconsistently defined across various branches of science. The statisticians’ strongest attacks are directed against work that differs from most laboratory experimental science. A weak point in the reasoning behind the Reproducibility Project and the statistical arguments is the assumption that a multi-pronged scientific investigation can be legitimately criticized by close examination of one of its components. Experimental science relies on multiple tests and multiple hypotheses to arrive at its conclusions. Reproducibility is a valid concern for science; it is not a “crisis.”

Publication bias, defined as a “tendency for positive results to be overrepresented in the published literature,” was recognized and bemoaned as early as the 17th century by the chemist Robert Boyle. In the latter half of the 20th century, it began to be recognized as an increasingly serious scientific problem characterized by a deluge of positive published results (actually exceeded 95% in some areas of psychology). And, by the second decade of the 21st century, data mining techniques indicated that the phenomenon had reached epic proportions, not only in psychology and the other social sciences, but in many of the life and physical sciences as well: a finding that might have been viewed as an amusing idiosyncratic scientific fact of life if not for a concomitant realization that most of these positive scientific findings were wrong. And that publication bias, if not a cause of this debacle, was at least a major facilitator. This chapter provides documentation for the high prevalence of this odd phenomenon in a wide swath of myriad empirical scientific literatures along with the accompanying compulsion it fosters for producing positive rather than reproducible results.

But what happens to investigators whose studies fails to replicate? The answer is complicated by the growing use of social media by scientists and the tenor of the original investigators’ responses to the replicators. Alternative case studies are presented including John Bargh’s vitriolic outburst following a failure of his classic word priming study to replicate, Amy Cuddy’s unfortunate experience with power posing, and Matthew Vees’s low-keyed response in which he declined to aggressively disparage his replicators, complemented the replicators’ interpretation of their replication, and neither defended his original study or even suggested that its findings might be wrong. In addition to such case studies, surveys on the subject suggest that there are normally no long-term deleterious career or reputational effects on investigators for a failure of a study to replicate and that a reasoned (or no) response to a failed replication is the superior professional and affective solution.