Skip to main content
All Reviews
PsychologyMust Read
beginner

Estimating the reproducibility of psychological science

Open Science Collaboration (2015)

Published
Aug 28, 2015
Journal
Science · Vol. 349 · No. 6251
View Original

At a Glance

100 replications, 5 indicators, 1 uncomfortable conclusion — psychology's published effects are substantially weaker than reported.

SummaryAI

270 researchers replicated 100 studies from three major psychology journals (PSCI, JPSP, JEP:LMC). Using five indicators of replication success: only 36% of replications achieved p < .05 (vs. 97% of originals); mean replication effect size was half the original; 47% of original effect sizes fell within the replication 95% CI; 39% were subjectively rated as replicated; and meta-analytically combining both studies left 68% significant.

​

Cognitive psychology effects replicated better than social (50% vs. 25% by significance). Replication success was predicted by strength of original evidence (lower p-values and larger effect sizes) rather than by team expertise or replication quality. The authors attribute much of the gap to publication and reporting biases inflating original estimates, and emphasize that the results don't establish any individual effect as true or false but reveal that the field's cumulative evidence base is less certain than assumed.

Method Snapshot

Pre-registered replications

Background

Basic statistics

In Collections

Psychology12 papers

The Reproducibility Crisis in Psychology

We are used to treating science as a source of knowledge that can be trusted a priori. But is this really the case? How much can we trust the results of psychological studies when making important decisions about our mental health and well-being. This collection contains articles devoted to checking psychological experiments — how well they can be replicated. In doing so, I examine two independent cohorts of studies: the reproducibility of the experiments themselves within large independent studies, as well as the reproducibility of meta-analyses. The latter are especially relevant, since the media often refer specifically to their conclusions.

ES
Evgeny Smirnov

Interesting results (poor reproducibility; weak effect), but the conclusions are overly cautious, possibly due to fear of the community’s reaction

— ES