Recalibrating single-study effect sizes using hierarchical Bayesian models

Zhipeng Cao; Matthew McCabe; Peter Callas; Renata B Cupertino; Jonatan Ottino-González; Alistair Murphy; Devarshi Pancholi; Nathan Schwab; Orr Catherine; Kent Hutchison; Janna Cousijn; Alain Dagher; John J Foxe; Anna E Goudriaan; Robert Hester; Chiang-Shan R Li; Wesley K Thompson; Angelica M Morales; Edythe D London; Valentina Lorenzetti; Maartje Luijten; Rocio Martin-Santos; Reza Momenan; Martin P Paulus; Lianne Schmaal; Rajita Sinha; Nadia Solowij; Dan J Stein; Elliot A Stein; Anne Uhlmann; Ruth J van Holst; Dick J Veltman; Reinout W Wiers; Murat Yücel; Sheng Zhang; Patricia Conrod; Scott Mackey; Hugh Garavan; ENIGMA Addiction Working Group

doi:10.3389/fnimg.2023.1138193

Recalibrating single-study effect sizes using hierarchical Bayesian models

Zhipeng Cao^*, Matthew McCabe, Peter Callas, Renata B Cupertino, Jonatan Ottino-González, Alistair Murphy, Devarshi Pancholi, Nathan Schwab, Orr Catherine, Kent Hutchison, Janna Cousijn, Alain Dagher, John J Foxe, Anna E Goudriaan, Robert Hester, Chiang-Shan R Li, Wesley K Thompson, Angelica M Morales, Edythe D London, Valentina LorenzettiMaartje Luijten, Rocio Martin-Santos, Reza Momenan, Martin P Paulus, Lianne Schmaal, Rajita Sinha, Nadia Solowij, Dan J Stein, Elliot A Stein, Anne Uhlmann, Ruth J van Holst, Dick J Veltman, Reinout W Wiers, Murat Yücel, Sheng Zhang, Patricia Conrod, Scott Mackey, Hugh Garavan, ENIGMA Addiction Working Group

^*Corresponding author for this work

Clinical Psychology

Research output: Contribution to journal › Article › Academic › peer-review

3 Citations (Scopus)

16 Downloads (Pure)

Abstract

INTRODUCTION:

There are growing concerns about commonly inflated effect sizes in small neuroimaging studies, yet no study has addressed recalibrating effect size estimates for small samples. To tackle this issue, we propose a hierarchical Bayesian model to adjust the magnitude of single-study effect sizes while incorporating a tailored estimation of sampling variance.

METHODS:

We estimated the effect sizes of case-control differences on brain structural features between individuals who were dependent on alcohol, nicotine, cocaine, methamphetamine, or cannabis and non-dependent participants for 21 individual studies (Total cases: 903; Total controls: 996). Then, the study-specific effect sizes were modeled using a hierarchical Bayesian approach in which the parameters of the study-specific effect size distributions were sampled from a higher-order overarching distribution. The posterior distribution of the overarching and study-specific parameters was approximated using the Gibbs sampling method.

RESULTS:

The results showed shrinkage of the posterior distribution of the study-specific estimates toward the overarching estimates given the original effect sizes observed in individual studies. Differences between the original effect sizes (i.e., Cohen's d) and the point estimate of the posterior distribution ranged from 0 to 0.97. The magnitude of adjustment was negatively correlated with the sample size (r = -0.27, p < 0.001) and positively correlated with empirically estimated sampling variance (r = 0.40, p < 0.001), suggesting studies with smaller samples and larger sampling variance tended to have greater adjustments.

DISCUSSION:

Our findings demonstrate the utility of the hierarchical Bayesian model in recalibrating single-study effect sizes using information from similar studies. This suggests that Bayesian utilization of existing knowledge can be an effective alternative approach to improve the effect size estimation in individual studies, particularly for those with smaller samples.

Original language	English
Article number	1138193
Journal	Frontiers in Neuroimaging
Volume	2
DOIs	https://doi.org/10.3389/fnimg.2023.1138193
Publication status	Published - 21 Dec 2023

Bibliographical note

Copyright © 2023 Cao, McCabe, Callas, Cupertino, Ottino-González, Murphy, Pancholi, Schwab, Catherine, Hutchison, Cousijn, Dagher, Foxe, Goudriaan, Hester, Li, Thompson, Morales, London, Lorenzetti, Luijten, Martin-Santos, Momenan, Paulus, Schmaal, Sinha, Solowij, Stein, Stein, Uhlmann, van Holst, Veltman, Wiers, Yücel, Zhang, Conrod, Mackey, Garavan and the ENIGMA Addiction Working Group.

Research programs

ESSB PSY

Access to Document

10.3389/fnimg.2023.1138193Licence: CC BY

Recalibrating single-study effect sizes using hierarchical Bayesian modelsFinal published version, 1.62 MBLicence: CC BY

Cite this

Cao, Z., McCabe, M., Callas, P., Cupertino, R. B., Ottino-González, J., Murphy, A., Pancholi, D., Schwab, N., Catherine, O., Hutchison, K., Cousijn, J., Dagher, A., Foxe, J. J., Goudriaan, A. E., Hester, R., Li, C.-S. R., Thompson, W. K., Morales, A. M., London, E. D., ... ENIGMA Addiction Working Group (2023). Recalibrating single-study effect sizes using hierarchical Bayesian models. Frontiers in Neuroimaging, 2, Article 1138193. https://doi.org/10.3389/fnimg.2023.1138193

@article{7a3f5fda90e742c1b364892c35304c0c,

title = "Recalibrating single-study effect sizes using hierarchical Bayesian models",

abstract = "INTRODUCTION: There are growing concerns about commonly inflated effect sizes in small neuroimaging studies, yet no study has addressed recalibrating effect size estimates for small samples. To tackle this issue, we propose a hierarchical Bayesian model to adjust the magnitude of single-study effect sizes while incorporating a tailored estimation of sampling variance.METHODS: We estimated the effect sizes of case-control differences on brain structural features between individuals who were dependent on alcohol, nicotine, cocaine, methamphetamine, or cannabis and non-dependent participants for 21 individual studies (Total cases: 903; Total controls: 996). Then, the study-specific effect sizes were modeled using a hierarchical Bayesian approach in which the parameters of the study-specific effect size distributions were sampled from a higher-order overarching distribution. The posterior distribution of the overarching and study-specific parameters was approximated using the Gibbs sampling method.RESULTS: The results showed shrinkage of the posterior distribution of the study-specific estimates toward the overarching estimates given the original effect sizes observed in individual studies. Differences between the original effect sizes (i.e., Cohen's d) and the point estimate of the posterior distribution ranged from 0 to 0.97. The magnitude of adjustment was negatively correlated with the sample size (r = -0.27, p < 0.001) and positively correlated with empirically estimated sampling variance (r = 0.40, p < 0.001), suggesting studies with smaller samples and larger sampling variance tended to have greater adjustments. DISCUSSION: Our findings demonstrate the utility of the hierarchical Bayesian model in recalibrating single-study effect sizes using information from similar studies. This suggests that Bayesian utilization of existing knowledge can be an effective alternative approach to improve the effect size estimation in individual studies, particularly for those with smaller samples.",

author = "Zhipeng Cao and Matthew McCabe and Peter Callas and Cupertino, \{Renata B\} and Jonatan Ottino-Gonz{\'a}lez and Alistair Murphy and Devarshi Pancholi and Nathan Schwab and Orr Catherine and Kent Hutchison and Janna Cousijn and Alain Dagher and Foxe, \{John J\} and Goudriaan, \{Anna E\} and Robert Hester and Li, \{Chiang-Shan R\} and Thompson, \{Wesley K\} and Morales, \{Angelica M\} and London, \{Edythe D\} and Valentina Lorenzetti and Maartje Luijten and Rocio Martin-Santos and Reza Momenan and Paulus, \{Martin P\} and Lianne Schmaal and Rajita Sinha and Nadia Solowij and Stein, \{Dan J\} and Stein, \{Elliot A\} and Anne Uhlmann and \{van Holst\}, \{Ruth J\} and Veltman, \{Dick J\} and Wiers, \{Reinout W\} and Murat Y{\"u}cel and Sheng Zhang and Patricia Conrod and Scott Mackey and Hugh Garavan and \{ENIGMA Addiction Working Group\}",

note = "Copyright {\textcopyright} 2023 Cao, McCabe, Callas, Cupertino, Ottino-Gonz{\'a}lez, Murphy, Pancholi, Schwab, Catherine, Hutchison, Cousijn, Dagher, Foxe, Goudriaan, Hester, Li, Thompson, Morales, London, Lorenzetti, Luijten, Martin-Santos, Momenan, Paulus, Schmaal, Sinha, Solowij, Stein, Stein, Uhlmann, van Holst, Veltman, Wiers, Y{\"u}cel, Zhang, Conrod, Mackey, Garavan and the ENIGMA Addiction Working Group.",

year = "2023",

month = dec,

day = "21",

doi = "10.3389/fnimg.2023.1138193",

language = "English",

volume = "2",

}

Cao, Z, McCabe, M, Callas, P, Cupertino, RB, Ottino-González, J, Murphy, A, Pancholi, D, Schwab, N, Catherine, O, Hutchison, K, Cousijn, J, Dagher, A, Foxe, JJ, Goudriaan, AE, Hester, R, Li, C-SR, Thompson, WK, Morales, AM, London, ED, Lorenzetti, V, Luijten, M, Martin-Santos, R, Momenan, R, Paulus, MP, Schmaal, L, Sinha, R, Solowij, N, Stein, DJ, Stein, EA, Uhlmann, A, van Holst, RJ, Veltman, DJ, Wiers, RW, Yücel, M, Zhang, S, Conrod, P, Mackey, S, Garavan, H & ENIGMA Addiction Working Group 2023, 'Recalibrating single-study effect sizes using hierarchical Bayesian models', Frontiers in Neuroimaging, vol. 2, 1138193. https://doi.org/10.3389/fnimg.2023.1138193

TY - JOUR

T1 - Recalibrating single-study effect sizes using hierarchical Bayesian models

AU - Cao, Zhipeng

AU - McCabe, Matthew

AU - Callas, Peter

AU - Cupertino, Renata B

AU - Ottino-González, Jonatan

AU - Murphy, Alistair

AU - Pancholi, Devarshi

AU - Schwab, Nathan

AU - Catherine, Orr

AU - Hutchison, Kent

AU - Cousijn, Janna

AU - Dagher, Alain

AU - Foxe, John J

AU - Goudriaan, Anna E

AU - Hester, Robert

AU - Li, Chiang-Shan R

AU - Thompson, Wesley K

AU - Morales, Angelica M

AU - London, Edythe D

AU - Lorenzetti, Valentina

AU - Luijten, Maartje

AU - Martin-Santos, Rocio

AU - Momenan, Reza

AU - Paulus, Martin P

AU - Schmaal, Lianne

AU - Sinha, Rajita

AU - Solowij, Nadia

AU - Stein, Dan J

AU - Stein, Elliot A

AU - Uhlmann, Anne

AU - van Holst, Ruth J

AU - Veltman, Dick J

AU - Wiers, Reinout W

AU - Yücel, Murat

AU - Zhang, Sheng

AU - Conrod, Patricia

AU - Mackey, Scott

AU - Garavan, Hugh

AU - ENIGMA Addiction Working Group

N1 - Copyright © 2023 Cao, McCabe, Callas, Cupertino, Ottino-González, Murphy, Pancholi, Schwab, Catherine, Hutchison, Cousijn, Dagher, Foxe, Goudriaan, Hester, Li, Thompson, Morales, London, Lorenzetti, Luijten, Martin-Santos, Momenan, Paulus, Schmaal, Sinha, Solowij, Stein, Stein, Uhlmann, van Holst, Veltman, Wiers, Yücel, Zhang, Conrod, Mackey, Garavan and the ENIGMA Addiction Working Group.

PY - 2023/12/21

Y1 - 2023/12/21

N2 - INTRODUCTION: There are growing concerns about commonly inflated effect sizes in small neuroimaging studies, yet no study has addressed recalibrating effect size estimates for small samples. To tackle this issue, we propose a hierarchical Bayesian model to adjust the magnitude of single-study effect sizes while incorporating a tailored estimation of sampling variance.METHODS: We estimated the effect sizes of case-control differences on brain structural features between individuals who were dependent on alcohol, nicotine, cocaine, methamphetamine, or cannabis and non-dependent participants for 21 individual studies (Total cases: 903; Total controls: 996). Then, the study-specific effect sizes were modeled using a hierarchical Bayesian approach in which the parameters of the study-specific effect size distributions were sampled from a higher-order overarching distribution. The posterior distribution of the overarching and study-specific parameters was approximated using the Gibbs sampling method.RESULTS: The results showed shrinkage of the posterior distribution of the study-specific estimates toward the overarching estimates given the original effect sizes observed in individual studies. Differences between the original effect sizes (i.e., Cohen's d) and the point estimate of the posterior distribution ranged from 0 to 0.97. The magnitude of adjustment was negatively correlated with the sample size (r = -0.27, p < 0.001) and positively correlated with empirically estimated sampling variance (r = 0.40, p < 0.001), suggesting studies with smaller samples and larger sampling variance tended to have greater adjustments. DISCUSSION: Our findings demonstrate the utility of the hierarchical Bayesian model in recalibrating single-study effect sizes using information from similar studies. This suggests that Bayesian utilization of existing knowledge can be an effective alternative approach to improve the effect size estimation in individual studies, particularly for those with smaller samples.

AB - INTRODUCTION: There are growing concerns about commonly inflated effect sizes in small neuroimaging studies, yet no study has addressed recalibrating effect size estimates for small samples. To tackle this issue, we propose a hierarchical Bayesian model to adjust the magnitude of single-study effect sizes while incorporating a tailored estimation of sampling variance.METHODS: We estimated the effect sizes of case-control differences on brain structural features between individuals who were dependent on alcohol, nicotine, cocaine, methamphetamine, or cannabis and non-dependent participants for 21 individual studies (Total cases: 903; Total controls: 996). Then, the study-specific effect sizes were modeled using a hierarchical Bayesian approach in which the parameters of the study-specific effect size distributions were sampled from a higher-order overarching distribution. The posterior distribution of the overarching and study-specific parameters was approximated using the Gibbs sampling method.RESULTS: The results showed shrinkage of the posterior distribution of the study-specific estimates toward the overarching estimates given the original effect sizes observed in individual studies. Differences between the original effect sizes (i.e., Cohen's d) and the point estimate of the posterior distribution ranged from 0 to 0.97. The magnitude of adjustment was negatively correlated with the sample size (r = -0.27, p < 0.001) and positively correlated with empirically estimated sampling variance (r = 0.40, p < 0.001), suggesting studies with smaller samples and larger sampling variance tended to have greater adjustments. DISCUSSION: Our findings demonstrate the utility of the hierarchical Bayesian model in recalibrating single-study effect sizes using information from similar studies. This suggests that Bayesian utilization of existing knowledge can be an effective alternative approach to improve the effect size estimation in individual studies, particularly for those with smaller samples.

UR - https://www.scopus.com/pages/publications/105005980914

U2 - 10.3389/fnimg.2023.1138193

DO - 10.3389/fnimg.2023.1138193

M3 - Article

C2 - 38179200

SN - 2813-1193

VL - 2

JO - Frontiers in Neuroimaging

JF - Frontiers in Neuroimaging

M1 - 1138193

ER -

Recalibrating single-study effect sizes using hierarchical Bayesian models

Abstract

Bibliographical note

Research programs

Access to Document

Other files and links

Fingerprint

Cite this