What is the probability of detecting poorly performing hospitals using funnel plots?

SE Seaton, L Barker, Hester Lingsma, Ewout Steyerberg, BN Manktelow

Research output: Contribution to journalArticleAcademicpeer-review

21 Citations (Scopus)


Recent high profile cases in the UK have highlighted the impact that the reporting of clinical outcomes can have for healthcare providers and patients. Great importance has been placed on the use of statistical methods to identify healthcare providers with observed poor performance. Such providers are highlighted as potential outliers and the possible causes investigated. It is crucial, therefore, that the methods used for identifying outliers are correctly understood and interpreted. While patients, funders, managers and clinical teams really want to know the true underlying performance of the provider, this true performance generally cannot be known directly and must be estimated using observed outcomes. However, differences between the true and observed performance are likely to arise due to chance variation. Clinical outcomes are often reported through the standardised mortality ratio (SMR), displayed using funnel plots. Providers whose observed SMR falls outside the funnel plot control limits will be identified as potential outliers. However, while it is obviously desirable that a healthcare provider with a true underlying performance different from that expected should be identified, the actual probability that it will be identified from observed SMRs has not previously been described. Here we show that funnel plots for the SMR should be used with caution when the expected number of events is small as the probability of identifying providers with true poor performance is likely to be small. On the other hand, when the expected number of events is large, care must be taken as a provider may be identified as an outlier even when its divergence is of little or no clinical importance.
Original languageUndefined/Unknown
Pages (from-to)870-876
Number of pages7
JournalBMJ Quality & Safety
Issue number10
Publication statusPublished - 2013

Research programs

  • EMC NIHES-02-65-01

Cite this