Bias, Precision and Timeliness of Historical (Background) Rate Comparison Methods for Vaccine Safety Monitoring: An Empirical Multi-Database Analysis

Xintong Li, Lana Y.H. Lai, Anna Ostropolets, Faaizah Arshad, Eng Hooi Tan, Paula Casajust, Thamir M. Alshammari, Talita Duarte-Salles, Evan P. Minty, Carlos Areia, Nicole Pratt, Patrick B. Ryan, George Hripcsak, Marc A. Suchard, Martijn J. Schuemie, Daniel Prieto-Alhambra

Research output: Contribution to journalArticleAcademicpeer-review

4 Citations (Scopus)
4 Downloads (Pure)


Using real-world data and past vaccination data, we conducted a large-scale experiment to quantify bias, precision and timeliness of different study designs to estimate historical background (expected) compared to post-vaccination (observed) rates of safety events for several vaccines. We used negative (not causally related) and positive control outcomes. The latter were synthetically generated true safety signals with incident rate ratios ranging from 1.5 to 4. Observed vs. expected analysis using within-database historical background rates is a sensitive but unspecific method for the identification of potential vaccine safety signals. Despite good discrimination, most analyses showed a tendency to overestimate risks, with 20%-100% type 1 error, but low (0% to 20%) type 2 error in the large databases included in our study. Efforts to improve the comparability of background and post-vaccine rates, including age-sex adjustment and anchoring background rates around a visit, reduced type 1 error and improved precision but residual systematic error persisted. Additionally, empirical calibration dramatically reduced type 1 to nominal but came at the cost of increasing type 2 error.

Original languageEnglish
Article number773875
JournalFrontiers in Pharmacology
Publication statusPublished - 24 Nov 2021

Bibliographical note

Funding Information:
UK National Institute of Health Research (NIHR), European Medicines Agency, Innovative Medicines Initiative 2 (806968), US Food and Drug Administration CBER BEST Initiative (75F40120D00039), and US National Library of Medicine (R01 LM006910). Australian National Health and Medical Research Council grant GNT1157506.

Publisher Copyright:
Copyright © 2021 Li, Lai, Ostropolets, Arshad, Tan, Casajust, Alshammari, Duarte-Salles, Minty, Areia, Pratt, Ryan, Hripcsak, Suchard, Schuemie and Prieto-Alhambra.


Dive into the research topics of 'Bias, Precision and Timeliness of Historical (Background) Rate Comparison Methods for Vaccine Safety Monitoring: An Empirical Multi-Database Analysis'. Together they form a unique fingerprint.

Cite this