Extracting the collective wisdom in probabilistic judgments

Ahmet Cem Peker

Research output: Contribution to journalArticleAcademicpeer-review

1 Citation (Scopus)


How should we combine disagreeing expert judgments on the likelihood of an event? A common solution is simple averaging, which allows independent individual errors to cancel out. However, judgments can be correlated due to an overlap in their information, resulting in a miscalibration in the simple average. Optimal weights for weighted averaging are typically unknown and require past data to estimate reliably. This paper proposes an algorithm to aggregate probabilistic judgments under shared information. Experts are asked to report a prediction and a meta-prediction. The latter is an estimate of the average of other individuals’ predictions. In a Bayesian setup, I show that if average prediction is a consistent estimator, the percentage of predictions and meta-predictions that exceed the average prediction should be the same. An “overshoot surprise” occurs when the two measures differ. The Surprising Overshoot algorithm uses the information revealed in an overshoot surprise to correct for miscalibration in the average prediction. Experimental evidence suggests that the algorithm performs well in moderate to large samples and in aggregation problems where individuals disagree in their predictions.

Original languageEnglish
Pages (from-to)467-501
Number of pages35
JournalTheory and Decision
Issue number3
Publication statusPublished - Apr 2023

Bibliographical note

Funding Information:
This work is supported by European Research Council Starting Grant 638408 Bayesian Markets.

Funding Information:
The author is grateful to Aurélien Baillon, Peter Wakker, Tom Wilkening, Han Bleichrodt, Ville Satopää and the anonymous referee for their comments. Special thanks to Tom Wilkening, Asa Palley and Ville Satopää for sharing their data sets. Comments from seminar participants at the ESA Job Market Seminars 2021, SJDM 2021 Annual meeting, Advances in Decision Analysis (ADA) 2022, MIT Human Cooperation Lab and the ESE Behavioral Economics group are much appreciated.

Publisher Copyright:
© 2022, The Author(s).


Dive into the research topics of 'Extracting the collective wisdom in probabilistic judgments'. Together they form a unique fingerprint.

Cite this