The principles on which the evidence-based movement operates are straightforward: fund programs with the best evidence of success, lavish attention on program implementation, evaluate outcomes, and change or replace programs if the evidence shows they don’t produce impacts. The attractiveness of these principles to analysts, policymakers, and program funders has been the basis of the emergence of a host of projects and activities devoted to implementing them. These programs include the Obama evidence-based initiative, which has created more than 1,400 local projects implementing model programs shown to produce good results in teen pregnancy prevention, home visiting, education, and several other areas of social policy;[1] a number of online clearinghouses that contain information about model programs that have been tested; the growth of Pay for Success programs, which now total more than 30 in the U.S. alone;[2] a host of federal activities driven by the Office of Management and Budget;[3] the Obama White House’s new research on applications of behavioral insights to increasing government efficiency;[4] and the Pew-MacArthur Results First Initiative. Of this impressive list of components of the evidence-based movement, which is by no means complete, the Results First initiative stands out for its emphasis on state and local policy.
Started in 2011, the Results First Initiative is now working with 19 states (about to be 20) and four counties to help them select and implement evidence-based programs in preschool education, K-12 education, juvenile and adult justice, home visiting, child protection, and other policy areas selected by states. Participating governments agree to follow several steps Pew-MacArthur has designed that can lead to the adoption of evidence-based policies by state and local decision-making bodies. Generalizing across several Pew-MacArthur publications, their unique process of working with states to develop evidence-based procedures leading to evidence-based policies can be broken down into several steps:
-
- States first make an inventory of all the programs they conduct in the policy area or areas they have decided to pursue, such as child protection, preschool programs, and juvenile and adult corrections;
- Using a remarkable web-based tool, called the “Results First Clearinghouse Database,” that Pew-MacArthur has developed for conducting a review of evidence-based programs across the contents of eight national clearinghouses, states assess the level of evidence that exists about the effectiveness of each program in their current inventory. The state can also identify other evidence-based programs that they could be using and compare them to their current programs;
- Using a sophisticated analytical model the Pew-MacArthur team has developed for conducing benefit-cost analyses, states calculate and compare the costs and benefits of both their current and alternative evidence-based programs. This allows the state to project the long-term benefits they could achieve by funding each program, and rank them by the return on investment each would generate for taxpayers and citizens;
- The analysis results are provided to agency leaders and the state legislature in a report that contains the evaluation and benefit-cost information for programs that the state is currently using and that the state could be using, leaving the decision-making about how money will be allocated to the legislative and administrative branches of state government where policy decisions belong;
- States are advised to carefully oversee implementation of the programs and ensure that they are implemented with fidelity to the model programs being used;
- States are also advised to collect outcome data on all programs and to implement rigorous evaluations of new programs.
The vision of Results First is to build the capacity within each state to carry out these tasks on an ongoing basis and use the results to improve the impacts of their programs.
To illustrate how the Pew-MacArthur approach works, I turn to an overview of how the program is operating to reduce adult recidivism in New Mexico.[5] The state organization that serves as the quarterback for the Results First project is the Legislative Finance Committee (LFC), a nonpartisan agency established in 1957 by the state legislature to handle issues and analyses related to the state budget. The LFC is governed by a Board composed of both Republicans and Democrats from the New Mexico House and Senate. The LFC regularly performs program evaluations. Given its role in the state budget, its program evaluation capacity, and its role of working from the inside of the legislative process with direct connections to both sides of the aisle in the New Mexico House and Senate, the LFC is a natural for coordinating and even driving the work of Results First. One of the first issues it decided to address as part of Results First was recidivism by adult prisoners.
Working with the New Mexico Corrections Department, the LFC found that of the prisoners released each year, about half are reconvicted within five years. A legislative report released in 2012 by the LFC estimated that recidivism among prisoners released in a given year would cost New Mexico taxpayers $360 million over the following 15 years. Not surprisingly, the legislature was intent on trying to do something about these costs, not just because of the possibility of savings accruing to the state budget, but also because of the costs imposed on the citizens of New Mexico by the crimes committed by released prisoners, estimated at around $40 million.
Working in accord with the Results First Initiative, the LFC found that New Mexico conducted more than 40 separate programs within the state’s prison facilities and that about 30 providers conducted programs for released prisoners to help them reenter society and prevent their return to prison. The focus of the LFC and the New Mexico Corrections Department was to spend state funds on programs with the best evidence of reducing recidivism. As an essential part of its Results First work, the LFC conducted an analysis of existing evidence-based programs designed to reduce recidivism. They found that New Mexico was funding eight evidence-based programs that served about 6,700 prisoners and ex-prisoners at a cost of around $9 million. The LFC also prepared a chart showing for each program how many prisoners and ex-prisoners were being served, current expenditures, the total benefits estimated by the Results First cost-benefit model, and the benefit-to-cost ratio. The state legislature and the New Mexico Corrections Department used this benefit-cost information to begin shifting funds to programs that produced higher benefit-cost ratios.
An interesting and potentially important outcome of New Mexico’s involvement with Results First has been an intense focus on program implementation. As the LFC states in its report: “We cannot stress enough that program fidelity – how well programs are implemented – is critically important to achieving the predicted outcomes.”[6] Thus, both the LFC and the Corrections Department are paying more attention to how the programs are being implemented by requiring “ongoing efforts to assess program delivery and, when necessary, taking corrective actions to hold program managers and providers accountable for program outcomes.”
Given that the actions described above were implemented in 2013, the New Mexico program has not yet produced enough information to determine whether recidivism rates are falling. Nonetheless, it is already clear that Results First has had a major impact on how New Mexico is trying to improve its programs designed to reduce recidivism. More specifically, New Mexico has summarized the intervention programs it is now using to reduce recidivism, conducted research to estimate the costs and benefits of their programs and other model programs that they could use, and actually changed the allocation of state dollars to support programs with more evidence of success. In addition, the state’s involvement with Results First has caused them, according to their own documents released by the powerful Legislative Finance Committee, to devote careful attention to program implementation. In short, it appears that New Mexico has applied all the major elements of the Results First Initiative summarized above to get a greater return on their investments to reduce recidivism by improving the quality of programs they employ and more carefully monitoring implementation. If the evidence-based movement in general, and the Results First Initiative in particular, are going to improve policies and programs, and even more important lead to reductions in the problems the policies and programs are designed to fight, the movement needs to reserve any claims of victory until those problems are actually reversed.
It is not the goal of Results First, or the evidence-based movement in general, to preempt the normal decision-making process by state legislatures and executive branch agencies. Rather, the goal is to increase the role of evidence in the decision-making process and to offer a new input – and one that is less biased than most inputs – to the process. The New Mexico example is hopeful because it demonstrates that evidence can be injected into the state decision-making process and result in legislative changes that reallocate resources. No one should expect that evidence will always trump other inputs to the policymaking process, but as the Pew-MacArthur Results First Initiative shows, the role of evidence in the allocation of public dollars can and should be expanded.
[1] Ron Haskins and Greg Margolis, Show Me the Evidence: Obama’s Fight for Rigor and Results in Social Policy (Washington, DC: Brookings, 2015).
[2] Nonprofit Finance Fund, “Pay for Success U.S. Activity,” available at http://www.payforsuccess.org/pay-success-deals-United-States.
[3] Office of Management and Budget, A Government of the Future, available at https://www.whitehouse.gov/sites/default/files/omb/budget/fy2016/assets/future.pdf.
[4] Executive Office of the President, “Social and Behavioral Science Team: Annual Report,” Washington, DC: National Science and Technology Council, September 2015.
[5] Legislative Finance Committee (New Mexico), “Results First: Evidence-Based Options to Improve Outcomes,” Santa Fe, NM: Author, July 2013.
[6] Ibid, p. 5.
