Estimating the reproducibility of psychological science

Open Science Collaboration

doi:10.1126/science.aac4716

Estimating the reproducibility of psychological science

Open Science Collaboration

Research output: Contribution to journal › Article › peer-review

5514 Citations (Scopus)

211 Downloads (Pure)

Abstract

Reproducibility is a defining feature of science, but the extent to which it characterizes current research is unknown. We conducted replications of 100 experimental and correlational studies published in three psychology journals using high-powered designs and original materials when available. Replication effects were half the magnitude of original effects, representing a substantial decline. Ninety-seven percent of original studies had statistically significant results. Thirty-six percent of replications had statistically significant results; 47% of original effect sizes were in the 95% confidence interval of the replication effect size; 39% of effects were subjectively rated to have replicated the original result; and if no bias in original results is assumed, combining original and replication results left 68% with statistically significant effects. Correlational tests suggest that replication success was better predicted by the strength of original evidence than by characteristics of the original and replication teams.

Original language	English
Article number	aac4716
Journal	Science
Volume	349
Issue number	6251
DOIs	https://doi.org/10.1126/science.aac4716
Publication status	Published - 28 Aug 2015

Bibliographical note

Please note this paper should be cited with the group author Open Science Collaboration. This is a collaboration of 270 authors (including Coventry University author Gavin Sullivan) who are all listed at the end of the paper.
This is the author's version of the work. It is posted here by the permission of the AAAS for personal use, not for redistribution. The definitive version was published in Science on 28 August 2015:
Vol. 349 no. 6251, DOI: 10.1126/science.aac4716 .

Keywords

reproducibility
psychological research
replication success

Access to Document

10.1126/science.aac4716

Post-PrintAccepted author manuscript, 788 KB

Cite this

@article{3552b0688d0d4dd9babdf82fe8704767,

title = "Estimating the reproducibility of psychological science",

abstract = "Reproducibility is a defining feature of science, but the extent to which it characterizes current research is unknown. We conducted replications of 100 experimental and correlational studies published in three psychology journals using high-powered designs and original materials when available. Replication effects were half the magnitude of original effects, representing a substantial decline. Ninety-seven percent of original studies had statistically significant results. Thirty-six percent of replications had statistically significant results; 47% of original effect sizes were in the 95% confidence interval of the replication effect size; 39% of effects were subjectively rated to have replicated the original result; and if no bias in original results is assumed, combining original and replication results left 68% with statistically significant effects. Correlational tests suggest that replication success was better predicted by the strength of original evidence than by characteristics of the original and replication teams.",

keywords = "reproducibility, psychological research, replication success",

author = "Sullivan, {Gavin B.} and {Open Science Collaboration}",

note = "Please note this paper should be cited with the group author Open Science Collaboration. This is a collaboration of 270 authors (including Coventry University author Gavin Sullivan) who are all listed at the end of the paper. This is the author's version of the work. It is posted here by the permission of the AAAS for personal use, not for redistribution. The definitive version was published in Science on 28 August 2015: Vol. 349 no. 6251, DOI: 10.1126/science.aac4716 .",

year = "2015",

month = aug,

day = "28",

doi = "10.1126/science.aac4716",

language = "English",

volume = "349",

journal = "Science",

issn = "0036-8075",

publisher = "American Association for the Advancement of Science",

number = "6251",

}

TY - JOUR

T1 - Estimating the reproducibility of psychological science

AU - Sullivan, Gavin B.

AU - Open Science Collaboration

N1 - Please note this paper should be cited with the group author Open Science Collaboration. This is a collaboration of 270 authors (including Coventry University author Gavin Sullivan) who are all listed at the end of the paper. This is the author's version of the work. It is posted here by the permission of the AAAS for personal use, not for redistribution. The definitive version was published in Science on 28 August 2015: Vol. 349 no. 6251, DOI: 10.1126/science.aac4716 .

PY - 2015/8/28

Y1 - 2015/8/28

N2 - Reproducibility is a defining feature of science, but the extent to which it characterizes current research is unknown. We conducted replications of 100 experimental and correlational studies published in three psychology journals using high-powered designs and original materials when available. Replication effects were half the magnitude of original effects, representing a substantial decline. Ninety-seven percent of original studies had statistically significant results. Thirty-six percent of replications had statistically significant results; 47% of original effect sizes were in the 95% confidence interval of the replication effect size; 39% of effects were subjectively rated to have replicated the original result; and if no bias in original results is assumed, combining original and replication results left 68% with statistically significant effects. Correlational tests suggest that replication success was better predicted by the strength of original evidence than by characteristics of the original and replication teams.

AB - Reproducibility is a defining feature of science, but the extent to which it characterizes current research is unknown. We conducted replications of 100 experimental and correlational studies published in three psychology journals using high-powered designs and original materials when available. Replication effects were half the magnitude of original effects, representing a substantial decline. Ninety-seven percent of original studies had statistically significant results. Thirty-six percent of replications had statistically significant results; 47% of original effect sizes were in the 95% confidence interval of the replication effect size; 39% of effects were subjectively rated to have replicated the original result; and if no bias in original results is assumed, combining original and replication results left 68% with statistically significant effects. Correlational tests suggest that replication success was better predicted by the strength of original evidence than by characteristics of the original and replication teams.

KW - reproducibility

KW - psychological research

KW - replication success

UR - http://science.sciencemag.org/content/suppl/2015/08/26/349.6251.aac4716.DC1

U2 - 10.1126/science.aac4716

DO - 10.1126/science.aac4716

M3 - Article

SN - 0036-8075

SN - 1095-9203

VL - 349

JO - Science

JF - Science

IS - 6251

M1 - aac4716

ER -

Estimating the reproducibility of psychological science

Abstract

Bibliographical note

Keywords

Access to Document

Other files and links

Fingerprint

Gavin Sullivan

Cite this

Estimating the reproducibility of psychological science

Abstract

Bibliographical note

Keywords

Access to Document

Other files and links

Fingerprint

Profiles

Gavin Sullivan

Cite this