Evaluation Design for Complex Global Initiatives

Every year, public and private funders spend many billions of dollars on large-scale, complex, multi-national health initiatives. The only way to know whether these initiatives are achieving their objectives is through evaluations that examine the links between program activities and desired outcomes. Investments in such evaluations, which, like the initiatives being evaluated, are carried out in some of the world's most challenging settings, are a relatively new phenomenon. In the last five years, evaluations have been conducted to determine the effects of some of the world's largest and most complex multi-national health initiatives. Evaluation Design for Complex Global Initiatives is the summary of a workshop convened by the Institute of Medicine in January 2014 to explore these recent evaluation experiences and to consider the lessons learned from how these evaluations were designed, carried out, and used. The workshop brought together more than 100 evaluators, researchers in the field of evaluation science, staff involved in implementing large-scale health programs, local stakeholders in the countries where the initiatives are carried out, policy makers involved in the initiatives, representatives of donor organizations, and others to derive lessons learned from past large-scale evaluations and to discuss how to apply these lessons to future evaluations. This report discusses transferable insights gained across the spectrum of choosing the evaluator, framing the evaluation, designing the evaluation, gathering and analyzing data, synthesizing findings and recommendations, and communicating key messages. The report also explores the relative benefits and limitations of different quantitative and qualitative approaches within the mixed methods designs used for these complex and costly evaluations. Table of ContentsFront Matter1 Introduction and Overview of the Workshop2 Overview Framework for Complex Evaluations3 Framing the Evaluation4 Developing the Evaluation Design and Selecting Methods5 Mapping Data Sources and Gathering and Assessing Data6 Applying Qualitative Methods to Evaluation on a Large Scale7 Applying Quantitative Methods to Evaluation on a Large Scale8 Analysis Through Triangulation and Synthesis to Interpret Data in a Mixed Methods Evaluation9 Evolving Methods in Evaluation Science10 Lessons from Large-Scale Program Evaluation on a Not-Quite-as-Large Scale11 Using Evaluation Findings and Communicating Key Messages12 Envisioning a Future for EvaluationReferencesAppendix A: Statement of TaskAppendix B: Workshop AgendaAppendix C: Participant BiographiesAppendix D: Evaluation Information Summary for Core Example InitiativesAppendix E: Evaluation Design Resources Highlighted at the Workshop