Measuring the Innovation Impact of Scientific Research: Exploring the Potentials of AI

Policy makers and funding agencies increasingly expect scientists to demonstrate the societal innovation impact of their knowledge production. But how can they assess the broader innovation impact of the research they provided funding for beyond the citation impact that publications associated with the funded research have? And, how can scientists increase their understanding in terms of how their research benefits society? In this paper, we explore the potential of machine learning to assess how scientific publications inform practice guidelines and as such shape actual decision-making in society. More specifically, we suggest an artificial intelligence-based semantic processing approach (AI) that seeks to identify the use of published research results in the field of diabetes in new clinical practice guidelines. Using this context allows to calibrate and validate the AI-based approach since guidelines contain citations to published articles while at the same time representing broader impact on clinical practice. Our results demonstrate the feasibility of AI-based impact measurement and identify a number of instances in which published research was used in guidelines but not cited (“hidden impact”) or cited but not actually used (“legitimacy building”). We follow up on these cases through a series of expert workshops and discussions to validate insights and identify potentials and limitations of an AI-based innovation impact measurement.