Progress and pitfalls when evaluating the unintended effects of public policy: The case of German international development cooperation

The study of unintended effects of policies is a key debate among evaluation scholars. Through complexity theory, we argue that unintended effects of (international) public actions are inevitable and question the reliability of evaluations in providing a correct and complete picture of public policy. We use a machine-learning-assisted text-mining case study approach, examining 254 programme evaluations of German international development cooperation as a ‘least likely case’. While German evaluations focus more on unintended effects than Dutch, Norwegian and American evaluations, their treatment is not always correct or complete. There is an overidentification of unintended effects and a bias towards positive ones, with certain types of unintended effects overlooked. We explore explanations for the observed weaknesses, including an overreliance on linear thinking and insufficient guidance for the evaluators on identifying unintended effects. We conclude with concrete suggestions to improve implementation of the Organization for Economic Co-operation and Development guidelines that are essential to make public administration more effective and trusted.