Skip to main content
All Reviews
PsychologyMust Read
beginner

Comparing meta-analyses and preregistered multiple-laboratory replication projects

Amanda Kvarven et al. (2019)

Published
Dec 23, 2019
Journal
Nature Human Behaviour · Vol. 4 · No. 4
DOI
10.1038/s41562-019-0787-z

At a Glance

Meta-analyses overestimate effect sizes 3x compared to multi-lab replications. Three times.

SummaryAI

The headline study of this sub-field. The authors matched 15 psychological effects that had both a published meta-analysis and a pre-registered multi-lab replication (from Many Labs, RRR, etc.). The mean meta-analytic effect size was 0.42; the mean replication effect size was 0.15 — nearly three times larger. The difference was significant for 12 of 15 pairs. Bias correction methods (trim-and-fill, PET-PEESE, 3PSM) reduced the gap somewhat but couldn't fully explain it. No evidence for "replicator selection" (that replicators systematically choose weak study designs).

Method Snapshot

Comparison between literature and independent estimations

Background

Basic statistics, but for the content, it's essential to check the Many Labs and OSC projects to get clarity on the independent verifications.

A good idea is to compare the conclusions from meta-analyses with the results of replication studies conducted by independent laboratories. The outcome, I suppose, is clear to you...

ES