4 types of data necessary for outcome-based financing

Children attend their first class immediately after they got registered at the school in Hazema North Raqqa, Syria August 21, 2017. Picture taken August 21, 2017. REUTERS/Zohra Bensemra - RC14C09853B0

The U.N. General Assembly is underway this week, and the global development community has descended upon New York en masse. With 11 years to achieve the ambitious Sustainable Development Goals (SDGs), some actors are looking to outcome-based financing as a way to ensure that funding is focused on results. Impact bonds are just one example of outcome-based financing, where private investors provide upfront capital, repaid conditional on the achievement of pre-determined metrics. Although evidence is sparse on the relative merits of this mechanism, millions of dollars are being invested in their design and implementation. Some of the challenges highlighted include lengthy design time and costly monitoring and evaluation.

In our fifth year of researching outcome-based financing, we are focusing on what is needed for these contracts to be executed efficiently and effectively, with the beneficiaries at the forefront. We have identified four types of data in particular that we think are necessary for the design and implementation of outcome-based financing:

  1. Cost data: How much does it actually cost to deliver desired outcomes?
  2. Cost of inaction data: How much would it cost government/society if the outcomes were not delivered?
  3. Real-time performance data: What is (not) working and for whom? What can be done?
  4. Results data: Are final outcomes being achieved?

Cost data

Price is often a determining factor in whether to purchase something. Service providers need cost data to make the case for increased investments. Even when convinced of the benefits, policymakers will want to know: How much will this program cost me? Cost data is also necessary for accountability and information about total expenditures, as well as for budgeting and estimating the financial implications of scaling-up services. Further, cost data is needed to perform cost-effectiveness and cost-benefit analyses, to aid decisionmaking on programs. However, in many instances there is little transparency about the costs of services, in particular social services. In outcome-based financing, it is critical to have accurate estimates of the cost of delivering services (and ideally outcomes), so they can feed into payment calculations. Our work on costing in the early childhood and education sectors attempts to bring clarity and consistency to cost calculations. In the coming year keep an eye out for our new user-friendly online tool for calculating costs of early childhood and education programs.

Cost of inaction data

The cost of inaction represents both direct and indirect costs to society if an intervention, which leads to desirable social or environmental outcomes, is not provided. For example, the indirect costs of not providing quality education would be poor economic growth as a result of low labor productivity and direct costs would be criminal justice system expenditures resulting from higher crime rates.

When financing is tied to specific outcomes, data on the cost of not achieving them helps funders to determine how much they should pay. In countries where there is a strong social safety net, direct costs avoided would include for example unemployment benefits, while in low-income countries, where such safety nets may not exist, the estimated value of an outcome may be mostly based on the opportunity costs. The cost of inaction is a central consideration in early childhood development, where missed opportunities may have severe individual and societal consequences. For example, recent estimates in the Lancet find that children at risk of poor development in low- and middle-income countries “are likely to lose one quarter of an average adult income per year which could account to as much as 2 times the GDP spent on health.”

Real-time data

A key feature of outcome-based financing is the need for service providers to use real-time data effectively to adapt their performance to meet targets and goals. This was a key success identified in Educate Girls—the first development impact bond (DIB) for education. Safeena Husain, CEO of Educate Girls, describes how the data and monitoring within the DIB supported frontline workers, and the focus on outcomes encouraged the organization to “think much more deeply about how that outcome is actually getting delivered.”

Bridges Fund Management, which has been involved in nearly 30-social impact bonds (SIBs) in the U.K., places an enormous emphasis on data: “We create the systems that allow us to collect data on positive and negative effects of the enterprise, and then use this information to pro-actively engage and inform strategy. We have learned that good governance, data-collection systems and a culture of learning are critical success factors to improving impact performance.”

In collecting real-time data, an organization needs to know what to collect and how often, as well as how to interpret the data and take action. However, service provider data capacity varies greatly, offering both a challenge and an opportunity for outcome-based financing. In some countries, systemic policy efforts have been dedicated to getting service providers ready to participate in outcome-based financing. For example, in the United States, the Social Innovation Fund provided grants to support capacity building and technical assistance for structuring Pay for Success projects.

Service providers also need to be able to collect data with accuracy and efficiently. Currently, both paper-based and technology-based data collection are being utilized in impact bonds, yet there are some draw backs to the paper-based collection method. First, it can be difficult to ensure that the data efficiently reaches decisionmakers; Second, it can be even more challenging to identify errors or trends in the data. As a result of these issues, determining payments in an impact bond is more challenging and potentially more costly. We have noted, however, that when technology is harnessed in impact bonds, often the tools are bespoke platforms developed specifically for the project, raising further questions around efficiency and cost. We believe that there must be a more efficient solution and are conducting research on digital tools for real-time data collection and feedback in the education sector—stay tuned for more on this topic down the road.

Results data

Finally, data related to results are at the heart of outcome-based financing. While ongoing data collection often informs service provider decisions, project success is typically framed around a handful of key metrics. Earlier iterations of results-based financing in international development typically relied on outputs to determine success and repayment. Outputs are metrics which come earlier in the results chain of a given intervention (Figure 1) and measure products which are within the control of the implementer; outcomes—and particularly final outcomes—are less tangible, and capture the overarching goals of the program.

Figure 1: Results chain

Results chain

Source: Gertler et al. 2016

Within impact bonds, payment metrics are typically toward the right-hand side of the results chain: measuring either intermediate or final outcomes.  According to analysis of the Brookings Global Impact Bonds Database (Table 1), the majority of impact bonds, or 65 percent of those with available data, use outcomes to determine payments.

Table 1: Metrics breakdown

Impact bond table

*Note: Calculations based on available data (n=157)

Nearly a third of projects use a mixture of metrics along the results chain, capturing both outputs (for example the number of beneficiaries who enter the program or receive the intervention), and outcomes (such as sustained employment). Just five projects use output data alone.

Judging whether metrics have been achieved will require data. For some metrics, administrative data may already be collected—for example on the employment status of participants. For others, new data will need to be gathered, and stakeholders will have to select an appropriate measurement tool. The tool will depend on the outcome(s) selected. In education impact bonds, where the final goal may be learning outcomes, this tool is likely to be an assessment—for example the Educate Girls DIB in India used the ASER assessment tool. In health impact bonds, metrics have been measured using tools for capturing wellbeing—for example the use of the well-being star in the U.K.’s Ways to Wellness SIB.

Tying payment to metrics later in the results chain is perhaps the main hypothesized benefit of outcome-based financing: coming closer to paying for the impact you want to achieve. Selecting the right metrics, and then identifying or collecting the data needed to measure them, are crucial components of paying for results.

In conclusion, from what we’ve observed thus far, contracting based on outcomes isn’t always easy and there is still much to be learned about the potential benefits as well as pitfalls of this model. We believe that the four types of data that we have laid out here are critical not only to outcome-based financing but also to any attempt to deliver on the promise of the SDGs. It’s possible impact bonds and their relatives are a Trojan horse. In the end, they are getting us to pay closer attention to the things that matter the most. We recognize however that data are a necessary but insufficient condition for achieving outcomes and that a richer understanding of human behavior and unmet social needs is paramount.

Note: Information related to the draw backs on the paper-based collection method were informed by a phone conversation with Joseph Di Silvio, the impact and performance manager at Volta Capital.