The power of impact evaluation for policy learning

Every year Colombia’s national tax collection agency, DIAN, sends out around 400 officials throughout the country to personally deliver notifications on delinquent obligations to thousands of taxpayers on a stampede-like, one-day effort. The “National Collection Day” was first implemented in 2009, and in 2013 the agency conducted an evaluation to assess the collection return of these visits compared to letters sent by either regular mail or email. 

The results showed that being contacted in person increased the probability of payment by over 90 percent, but receiving an email or a letter increased the probability of payment by around 20 percent; however, it is much easier to reach people by email (88 percent success rate) than physically at their homes (25 percent success rate). These results led the authorities to target visits first on the quality of the taxpayer’s contact information and then on debt size, which increased their visit success rate to over 85 percent.

There are two key messages here: First, a fairly simple impact evaluation led to actionable answers through which the organization was able to improve its performance. Second, management within the organization remains enthusiastic about using impact evaluation as a tool for decision-making, leading to a number of additional internal evaluation projects. This is promising for the future of impact evaluation.

There is certainly a role for impact evaluation in improving our understanding of the way society functions and the specific weight certain behavioral theories carry in our worldview; however, the tools that allow us to ascertain causal relationships between actions and outcomes need to become a part of the public decision-making process and owned by public administrators just as any other tool. While not every decision over public resources can be piloted and evaluated with scientific credibility, there is certainly always the potential to learn.

The greatest significance of policy learning evaluations lies beyond the evaluation itself; it lies in the process whereby institutions recognize the value of the learning experience and subsequently choose to apply these lessons to other initiatives. The key principle is to focus on the learning experience, with a commitment to scientific rigor instead of to a particular evaluation methodology. Institutions that recognize the educational value of their experience are the most adept at transforming experience in to knowledge and thus transforming pitfalls into opportunities for better use of resources. This is the most powerful capacity building initiative the development community can possibly support.

Can we set a new standard for impact evaluation?

Whether an evaluation leads to a published academic article is generally viewed as the seal of a contribution to society. The rigors of peer review and subsequent publication of the research is one of the foundations of modern human progress; however, the way in which society transforms such knowledge into better choices is jumbled and sluggish. It is in no way clear that these publications provide timely or relevant answers to policymakers’ questions. So, it’s fair to ask that, instead of leading to academic papers, we value impact evaluations by their influence on policy and, most importantly, on institutions.

Is it possible to measure an evaluation’s impact in this sense? A rigorous evaluation of the state capacity effects of impact evaluations would be complicated by obvious selection bias. However, it is possible to measure results and to document the institutions’ progress toward a culture of learning. This is what Development Bank of Latin America (CAF) has set out to do around its own impact evaluations, going back to the partner institutions at least a year after the final impact evaluation report and asking questions about additional evaluation initiatives, communication efforts, and perceived influence in policy decisions.

Evaluating the ‘how’ instead of the ‘what’

What policies get implemented is the result of a complex mixture of institutional inertia, ideology (beliefs about what is good for society), scientific knowledge, and special interests. One of the reasons why policymakers often resist evaluation is the risk of harm to their political capital; a negative or not-so-positive result might signal corruption or worse yet, incompetence. This is not restricted to elected officials; it applies just as well to managers who seek to promotions within the public sector, and where altering the status quo might not be valued. Lowering the barriers to evaluation requires a look at public policies where the political cost to evaluating is initially very low.

CAF proposed an emphasis on evaluating the impact of different ways of delivering certain policies, or implementing initiatives. The tax collection evaluation from Colombia is an example: It is not an evaluation of whether taxes are higher or lower but is instead an evaluation of how they are collected.

In partnership with CAF, the National Planning Department of Colombia, launched a call for “public management” evaluations, whereby agencies would propose management initiatives that they wish to have evaluated by our partnership. Over 60 proposals were received, and among those selected was the communications strategy of the Colombian Ministry of Culture with local governments across the country to make more efficient use of resources assigned to them. Another project asked about the most effective way to entice top quality teachers to work in underserved regions of the country.  And another initiative sought to bridge the communications gap between an employment agency and the unemployed and underemployed population.

The road to development requires better decisions about public resources, and what ‘better’ means is not always known, even with overriding good intentions. So we need institutions capable of transforming their experience into knowledge. The basis of our survival as a species has been our ability to adapt and transform our environment and our actions to better fit the circumstances. It seems natural that the basis of our sustained progress is also the capacity of our institutions to emulate the most powerful trait of the human condition: our capacity to learn.