Skip to main content
All Reviews
PsychologySkim
beginner

Reproducibility of Published Meta-Analyses on Clinical-Psychological Interventions

Rubén López-Nicolás et al. (2024)

Published
Jan 1, 2024
Journal
Advances in Methods and Practices in Psychological Science · Vol. 7 · No. 1
DOI
10.1177/25152459231202929
View OriginalView PDF

At a GlanceAI

Even clinical meta-analyses (the ones that guide actual treatment) are only 67% reproducible.

Summary

From 100 randomly selected articles on clinical-psychological interventions (2000–2020), 217 meta-analyses were evaluated. Only 67% were "process reproducible" (the data could even be retrieved). Of those, 52 showed discrepancies > 5% in main results. After multi-stage correction (fixing coding errors, qualitative assessment, contacting authors), 27 meta-analyses from 10 papers remained irreproducible. The process-reproducible rate improved over time (41% for 2000–2010, 80% for 2016–2020), suggesting data-sharing norms are working. Most numerical discrepancies were minor and didn't change conclusions — but data availability remains the biggest barrier.

Method Snapshot

random selection multi-stage reproducibility audit

Background

common sense

In Collections

Psychology12 papers

The Reproducibility Crisis in Psychology

We are used to treating science as a source of knowledge that can be trusted a priori. But is this really the case? How much can we trust the results of psychological studies when making important decisions about our mental health and well-being. This collection contains articles devoted to checking psychological experiments — how well they can be replicated. In doing so, I examine two independent cohorts of studies: the reproducibility of the experiments themselves within large independent studies, as well as the reproducibility of meta-analyses. The latter are especially relevant, since the media often refer specifically to their conclusions.

ES
Evgeny Smirnov

The authors rightly note that the key problem is the lack of access to the original raw data, plus the fact that if a paper says “data upon request,” it usually means “no.” I agree 100%.

— ES