Many Labs 2: Investigating Variation in Replicability Across Samples and Settings

  • Richard A. Klein (Université Grenoble Alpes) (Creator)
  • Michelangelo Vianello (University of Padua) (Creator)
  • Fred Hasselman (Radboud University Nijmegen) (Creator)
  • Byron G. Adams (University of Johannesburg, Tilburg University) (Creator)
  • Reginald B. Adams (Pennsylvania State University) (Creator)
  • Sinan Alper (Yasar University) (Creator)
  • Mark Aveyard (American University of Sharjah) (Creator)
  • Jordan R. Axt (Duke University) (Creator)
  • Mayowa T. Babalola (United Arab Emirates University) (Contributor)
  • Štěpán Bahník (University of Economics, Prague) (Contributor)
  • Rishtee Batra (Saint Josephs University) (Contributor)
  • Mihály Berkics (Eotvos Lorand University) (Creator)
  • Michael J. Bernstein (Pennsylvania State University) (Creator)
  • Daniel R. Berry (California State University San Marcos) (Creator)
  • Olga Bialobrzeska (SWPS University of Social Sciences and Humanities) (Creator)
  • Evans Dami Binan (University of Jos) (Contributor)
  • Konrad Bocian (SWPS University of Social Sciences and Humanities) (Creator)
  • Mark J. Brandt (Tilburg University) (Creator)
  • Robert Busching (University of Potsdam) (Creator)
  • Anna Cabak Rédei (Lund University) (Creator)
  • Huajian Cai (Chinese Academy of Sciences) (Contributor)
  • Fanny Cambier (Université catholique de Louvain) (Creator)
  • Katarzyna Cantarero (SWPS University of Social Sciences and Humanities) (Creator)
  • Cheryl L. Carmichael (City University of New York) (Creator)
  • Francisco Ceric (Universidad del Desarrollo) (Creator)
  • Jesse Chandler (Mathematica Policy Research Inc., University of Michigan, Ann Arbor) (Creator)
  • Jen-Ho Chang (National Taiwan University, Academia Sinica - Institute of Ethnology) (Contributor)
  • Armand Chatard (Université de Poitiers, CNRS Unité Mixte de Recherche 7295) (Creator)
  • Eva E. Chen (Hong Kong University of Science and Technology) (Creator)
  • Winnee Cheong (HELP University) (Contributor)
  • David C. Cicero (University of Hawai'i at Mānoa) (Creator)
  • Sharon Coen (University of Salford) (Creator)
  • Jennifer A. Coleman (Virginia Commonwealth University) (Creator)
  • Brian Collisson (Azusa Pacific University) (Creator)
  • Morgan A. Conway (University of Florida) (Creator)
  • Katherine S. Corker (Grand Valley State University) (Creator)
  • Paul G. Curran (Grand Valley State University) (Creator)
  • Fiery Cushman (Harvard University) (Contributor)
  • Zubairu K. Dagona (University of Jos) (Contributor)
  • Ilker Dalgar (Middle East Technical University) (Contributor)
  • Anna Dalla Rosa (University of Padua) (Contributor)
  • William E. Davis (Wittenberg University) (Creator)
  • Maaike de Bruijn (Tilburg University) (Contributor)
  • Leander De Schutter (Contributor)
  • Thierry Devos (San Diego State University) (Creator)
  • Marieke de Vries (Radboud University Nijmegen, Tilburg University) (Contributor)
  • Canay Doğulu (Baskent University) (Contributor)
  • Nerisa Dozo (University of Queensland) (Contributor)
  • Kristin Nicole Dukes (Allegheny College) (Creator)
  • Yarrow Dunham (Yale University) (Contributor)
  • Kevin Durrheim (University of KwaZulu-Natal) (Creator)
  • Charles R. Ebersole (University of Virginia) (Creator)
  • John E. Edlund (Rochester Institute of Technology) (Creator)
  • Anja Eller (Universidad Nacional Autónoma de México) (Creator)
  • Alexander Scott English (Shanghai International Studies University) (Creator)
  • Carolyn Finck (Universidad de los Andes Colombia) (Creator)
  • Natalia Frankowska (SWPS University of Social Sciences and Humanities) (Creator)
  • Miguel-Ángel Freyre (Universidad Nacional Autónoma de México) (Contributor)
  • Mike Friedman (Université catholique de Louvain) (Creator)
  • Elisa Maria Galliani (University of Padua) (Creator)
  • Joshua C. Gandi (University of Jos) (Creator)
  • Tanuka Ghoshal (City University of New York) (Contributor)
  • Steffen Giessner (Creator)
  • Tripat Gill (Lazaridis School of Business and Economics) (Contributor)
  • Timo Gnambs (Johannes Kepler University Linz, Leibniz Institute for Educational Trajectories) (Creator)
  • Ángel Gómez (National Distance Education University) (Creator)
  • Roberto González (Pontificia Universidad Católica de Chile) (Creator)
  • Jesse Graham (University of Utah) (Creator)
  • Jon E. Grahe (Pacific Lutheran University) (Creator)
  • Ivan Grahek (Ghent University) (Creator)
  • Eva G. T. Green (University of Lausanne) (Creator)
  • Kakul Hai (Amity University) (Contributor)
  • Matthew Haigh (Northumbria University) (Creator)
  • Elizabeth L. Haines (William Paterson University) (Creator)
  • Michael P. Hall (University of Michigan, Ann Arbor) (Creator)
  • Marie E. Heffernan (Children's Memorial Hospital) (Creator)
  • Joshua A. Hicks (Texas A&M University) (Creator)
  • Petr Houdek (Jan Evangelista Purkyne University in Usti nad Labem) (Creator)
  • Jeffrey R. Huntsinger (Loyola University Chicago) (Creator)
  • Ho Phi Huynh (Texas A&M University-San Antonio) (Creator)
  • Hans Ijzerman (Université Grenoble Alpes) (Contributor)
  • Yoel Inbar (University of Toronto) (Creator)
  • Åse H. Innes-Ker (Lund University) (Creator)
  • William Jiménez-Leal (Universidad de los Andes Colombia) (Creator)
  • Melissa-Sue John (Worcester Polytechnic Institute) (Contributor)
  • Jennifer A. Joy-Gaba (Virginia Commonwealth University) (Creator)
  • Roza G. Kamiloğlu (University of Amsterdam) (Creator)
  • Heather Barry Kappes (The London School of Economics and Political Science) (Creator)
  • Serdar Karabati (Istanbul Bilgi University) (Creator)
  • Haruna Karick (SWPS University of Social Sciences and Humanities, University of Jos) (Creator)
  • Victor N. Keller (Universidade de Brasília) (Creator)
  • Anna Kende (Eotvos Lorand University) (Creator)
  • Nicolas Kervyn (Université catholique de Louvain) (Creator)
  • Goran Knežević (University of Belgrade) (Creator)
  • Carrie Kovacs (Johannes Kepler University Linz) (Creator)
  • Lacy E. Krueger (Texas A&M University-Commerce) (Creator)
  • German Kurapov (Tilburg University) (Creator)
  • Jamie Kurtz (James Madison University) (Creator)
  • Daniël Lakens (Eindhoven University of Technology) (Creator)
  • Ljiljana B. Lazarević (University of Belgrade) (Creator)
  • Carmel A. Levitan (Occidental College) (Creator)
  • Neil A. Lewis (Cornell University) (Creator)
  • Samuel Lins (University of Porto) (Creator)
  • Nikolette P. Lipsey (University of Florida) (Contributor)
  • Joy E. Losee (University of Florida) (Creator)
  • Esther Maassen (Tilburg University) (Creator)
  • Angela T. Maitner (American University of Sharjah) (Creator)
  • Winfrida Malingumu (Open University of Tanzania) (Contributor)
  • Robyn K. Mallett (Loyola University Chicago) (Creator)
  • Satia A. Marotta (Tufts University) (Contributor)
  • Janko Međedović (Institute of Criminological and Sociological Research, Singidunum University) (Creator)
  • Fernando Mena-Pacheco (Universidad Latina de Costa Rica) (Creator)
  • Taciano L. Milfont (Victoria University of Wellington) (Contributor)
  • Wendy L. Morris (McDaniel College) (Creator)
  • Sean C. Murphy (University of Melbourne) (Creator)
  • Andriy Myachykov (Northumbria University) (Creator)
  • Nick Neave (Northumbria University) (Creator)
  • Koen Neijenhuijs (University of Amsterdam, Vrije Universiteit Amsterdam) (Creator)
  • Anthony J. Nelson (Pennsylvania State University) (Creator)
  • Félix Neto (University of Porto) (Creator)
  • Austin Lee Nichols (University of Central Florida) (Creator)
  • Aaron Ocampo (Universidad Latina de Costa Rica) (Creator)
  • Susan L. O’donnell (George Fox University) (Contributor)
  • Haruka Oikawa (Doshisha University) (Creator)
  • Masanori Oikawa (Doshisha University) (Creator)
  • Elsie Ong (Hong Kong Metropolitan University) (Creator)
  • Gábor Orosz (Stanford University) (Creator)
  • Malgorzata Osowiecka (SWPS University of Social Sciences and Humanities) (Contributor)
  • Grant Packard (Lazaridis School of Business and Economics) (Creator)
  • Rolando Pérez-Sánchez (University of Costa Rica) (Creator)
  • Boban Petrović (Institute of Criminological and Sociological Research) (Creator)
  • Ronaldo Pilati (Universidade de Brasília) (Creator)
  • Brad Pinter (Pennsylvania State University) (Creator)
  • Lysandra Podesta (Radboud University Nijmegen) (Creator)
  • Gabrielle Pogge (University of Florida) (Creator)
  • Monique M. H. Pollmann (Tilburg University) (Creator)
  • Abraham M. Rutchick (California State University Northridge) (Creator)
  • Patricio Saavedra (University of Sussex) (Creator)
  • Alexander K. Saeri (Monash University) (Creator)
  • Erika Salomon (The University of Chicago) (Creator)
  • Kathleen Schmidt (Southern Illinois University) (Creator)
  • Felix D. Schönbrodt (Ludwig Maximilian University of Munich) (Creator)
  • Maciej B. Sekerdej (Jagiellonian University in Kraków) (Creator)
  • David Sirlopú (Universidad del Desarrollo) (Creator)
  • Jeanine L. M. Skorinko (Worcester Polytechnic Institute) (Creator)
  • Michael A. Smith (Northumbria University) (Creator)
  • Vanessa Smith-Castro (University of Costa Rica) (Creator)
  • Karin C.H.J. Smolders (Creator)
  • Agata Sobkow (SWPS University of Social Sciences and Humanities) (Creator)
  • Walter Sowden (Walter Reed Army Institute of Research) (Creator)
  • Philipp Spachtholz (Ludwig Maximilian University of Munich) (Creator)
  • Manini Srivastava (University of Regensburg) (Contributor)
  • Troy G. Steiner (Pennsylvania State University) (Creator)
  • Jeroen Stouten (KU Leuven) (Creator)
  • Chris N. H. Street (University of Huddersfield) (Creator)
  • Oskar K. Sundfelt (Lund University) (Creator)
  • Stephanie Szeto (University of Salford) (Creator)
  • Ewa Szumowska (Jagiellonian University in Kraków) (Creator)
  • Andrew C. W. Tang (Hong Kong Metropolitan University) (Creator)
  • Norbert Tanzer (University of Graz) (Creator)
  • Morgan J. Tear (Monash University) (Creator)
  • Jordan Theriault (Northeastern University) (Creator)
  • Manuela Thomae (University of Winchester) (Creator)
  • David Torres (Universidad de Iberoamerica) (Creator)
  • Jakub Traczyk (SWPS University of Social Sciences and Humanities) (Creator)
  • Joshua M. Tybur (Vrije Universiteit Amsterdam) (Creator)
  • Adrienn Ujhelyi (Eotvos Lorand University) (Creator)
  • Robbie C. M. van Aert (Tilburg University) (Contributor)
  • Marcel A. L. M. van Assen (Tilburg University) (Contributor)
  • Marije van der Hulst (Contributor)
  • Paul A. M. van Lange (Vrije Universiteit Amsterdam) (Contributor)
  • Anna Elisabeth van ’t Veer (Contributor)
  • Alejandro Vásquez-Echeverría (Universidad de la República) (Contributor)
  • Leigh Ann Vaughn (Ithaca College) (Creator)
  • Alexandra Vázquez (National Distance Education University) (Creator)
  • Luis Diego Vega (Universidad Latina de Costa Rica) (Creator)
  • Catherine Verniers (Université Paris Cité) (Creator)
  • Mark Verschoor (University of Groningen) (Creator)
  • Ingrid P. J. Voermans (Radboud University Nijmegen) (Creator)
  • Marek A. Vranka (Charles University) (Creator)
  • Cheryl Welch (James Madison University) (Creator)
  • Aaron L. Wichman (Western Kentucky University) (Creator)
  • Lisa A. Williams (University of New South Wales) (Creator)
  • Michael Wood (University of Winchester) (Creator)
  • Julie A. Woodzicka (Washington and Lee University) (Creator)
  • Marta K. Wronska (SWPS University of Social Sciences and Humanities) (Creator)
  • Liane Young (Boston College) (Creator)
  • John M. Zelenski (Carleton University) (Creator)
  • Zeng Zhijia (Zhejiang University of Finance and Economics) (Creator)
  • Brian A. Nosek (University of Virginia, Center for Open Science) (Creator)

Dataset

Description

We conducted preregistered replications of 28 classic and contemporary published findings, with protocols that were peer reviewed in advance, to examine variation in effect magnitudes across samples and settings. Each protocol was administered to approximately half of 125 samples that comprised 15,305 participants from 36 countries and territories. Using the conventional criterion of statistical significance (p < .05), we found that 15 (54%) of the replications provided evidence of a statistically significant effect in the same direction as the original finding. With a strict significance criterion (p < .0001), 14 (50%) of the replications still provided such evidence, a reflection of the extremely high-powered design. Seven (25%) of the replications yielded effect sizes larger than the original ones, and 21 (75%) yielded effect sizes smaller than the original ones. The median comparable Cohen’s ds were 0.60 for the original findings and 0.15 for the replications. The effect sizes were small (< 0.20) in 16 of the replications (57%), and 9 effects (32%) were in the direction opposite the direction of the original effect. Across settings, the Q statistic indicated significant heterogeneity in 11 (39%) of the replication effects, and most of those were among the findings with the largest overall effect sizes; only 1 effect that was near zero in the aggregate showed significant heterogeneity according to this measure. Only 1 effect had a tau value greater than .20, an indication of moderate heterogeneity. Eight others had tau values near or slightly above .10, an indication of slight heterogeneity. Moderation tests indicated that very little heterogeneity was attributable to the order in which the tasks were performed or whether the tasks were administered in lab versus online. Exploratory comparisons revealed little heterogeneity between Western, educated, industrialized, rich, and democratic (WEIRD) cultures and less WEIRD cultures (i.e., cultures with relatively high and low WEIRDness scores, respectively). Cumulatively, variability in the observed effect sizes was attributable more to the effect being studied than to the sample or setting in which it was studied.
Date made available2018
  • Many labs 2: Investigating variation in replicability across samples and settings

    Klein, R. A., Vianello, M., Hasselman, F., Adams, B. G., Adams, R. B., Alper, S., Aveyard, M., Axt, J. R., Babalola, M. T., Bahník, Š., Batra, R., Berkics, M., Bernstein, M. J., Berry, D. R., Bialobrzeska, O., Binan, E. D., Bocian, K., Brandt, M. J., Busching, R., Rédei, A. C., & 170 othersCai, H., Cambier, F., Cantarero, K., Carmichael, C. L., Ceric, F., Chandler, J., Chang, J. H., Chatard, A., Chen, E. E., Cheong, W., Cicero, D. C., Coen, S., Coleman, J. A., Collisson, B., Conway, M. A., Corker, K. S., Curran, P. G., Cushman, F., Dagona, Z. K., Dalgar, I., Dalla Rosa, A., Davis, W. E., de Bruijn, M., De Schutter, L., Devos, T., de Vries, M., Doğulu, C., Dozo, N., Dukes, K. N., Dunham, Y., Durrheim, K., Ebersole, C. R., Edlund, J. E., Eller, A., English, A. S., Finck, C., Frankowska, N., Freyre, M. Á., Friedman, M., Galliani, E. M., Gandi, J. C., Ghoshal, T., Giessner, S. R., Gill, T., Gnambs, T., Gómez, Á., González, R., Graham, J., Grahe, J. E., Grahek, I., Green, E. G. T., Hai, K., Haigh, M., Haines, E. L., Hall, M. P., Heffernan, M. E., Hicks, J. A., Houdek, P., Huntsinger, J. R., Huynh, H. P., Ijzerman, H., Inbar, Y., Innes-Ker, Å. H., Jiménez-Leal, W., John, M. S., Joy-Gaba, J. A., Kamiloğlu, R. G., Kappes, H. B., Karabati, S., Karick, H., Keller, V. N., Kende, A., Kervyn, N., Knežević, G., Kovacs, C., Krueger, L. E., Kurapov, G., Kurtz, J., Lakens, D., Lazarević, L. B., Levitan, C. A., Lewis, N. A., Lins, S., Lipsey, N. P., Losee, J. E., Maassen, E., Maitner, A. T., Malingumu, W., Mallett, R. K., Marotta, S. A., Međedović, J., Mena-Pacheco, F., Milfont, T. L., Morris, W. L., Murphy, S. C., Myachykov, A., Neave, N., Neijenhuijs, K., Nelson, A. J., Neto, F., Nichols, A. L., Ocampo, A., O’donnell, S. L., Oikawa, H., Oikawa, M., Ong, E., Orosz, G., Osowiecka, M., Packard, G., Pérez-Sánchez, R., Petrović, B., Pilati, R., Pinter, B., Podesta, L., Pogge, G., Pollmann, M. M. H., Rutchick, A. M., Saavedra, P., Saeri, A. K., Salomon, E., Schmidt, K., Schönbrodt, F. D., Sekerdej, M. B., Sirlopú, D., Skorinko, J. L. M., Smith, M. A., Smith-Castro, V., Smolders, K. C. H. J., Sobkow, A., Sowden, W., Spachtholz, P., Srivastava, M., Steiner, T. G., Stouten, J., Street, C. N. H., Sundfelt, O. K., Szeto, S., Szumowska, E., Tang, A. C. W., Tanzer, N., Tear, M. J., Theriault, J., Thomae, M., Torres, D., Traczyk, J., Tybur, J. M., Ujhelyi, A., van Aert, R. C. M., van Assen, M. A. L. M., van der Hulst, M., van Lange, P. A. M., van ’t Veer, A. E., Vásquez-Echeverría, A., Vaughn, L. A., Vázquez, A., Vega, L. D., Verniers, C., Verschoor, M., Voermans, I. P. J., Vranka, M. A., Welch, C., Wichman, A. L., Williams, L. A., Wood, M., Woodzicka, J. A., Wronska, M. K., Young, L., Zelenski, J. M., Zhijia, Z. & Nosek, B. A., 2018, In: Advances in Methods and Practices in Psychological Science. 1, 4, p. 443-490 48 p.

    Research output: Contribution to journalArticleAcademicpeer-review

    Open Access
    File
    525 Citations (Scopus)
    25 Downloads (Pure)

Cite this