Short-Sighted Challenges to Evidence-Based Policy

Something revolutionary is happening: Evidence-based policy is taking hold in Washington. As described in two new books – Moneyball for Government and Show Me the Evidence – the Obama administration has made a series of strategic moves to increase the use of evidence by federal agencies and state and local programs. In other words, these programs are being assessed to see if they actually work – not a common occurrence inside the Beltway. 

These efforts build on similar steps initiated in the Bush administration. First, many federal agencies are using new competitive grant programs to award funds to organizations if they adopt program models and practices (“interventions”) that have rigorous evidence of success – and then to carefully evaluate the interventions to find out if they are working and if not, to make changes to improve them. In this way, more and more federal grants will employ taxpayer dollars to support interventions that have strong evidence of success.

This approach does not address the hundreds of pre-existing federal grant programs that together spend tens of billions of dollars on interventions that either have not been evaluated or, if evaluated, have little or no evidence of success. Thus, the second element of the Administration’s strategy is to change the way these existing grant programs allocate their dollars to encourage the rigorous evaluation of existing interventions as well as innovative new interventions to grow the number that are proven effective. The goal is to use rigorous evidence to evolve these existing programs toward greater effectiveness over time.  Sound controversial?  It shouldn’t be.  But it is – because Washington tends to run on financial interests and political ideology.

A recent episode shows the challenges in making even modest improvements in grant programs in which the money has already been allocated to state or local providers.

The program in question is Student Support Services, administered by the U.S. Department of Education. In operation since 1968, Student Support Services provides academic tutoring, counseling, and financial aid information to around 200,000 disadvantaged students at about 1,000 colleges and universities once they arrive on campus. Survey data show that these students have extraordinarily high college dropout rates, so programs like Student Support Services have the potential to fill an important purpose.

The Department recently proposed a modest change in the program’s funding criteria to encourage grant applicants to adopt interventions that have evidence of effectiveness. What a concept!  As the Department’s draft notice to applicants, issued on September 29, puts the matter: “We believe that encouraging applicants to focus on proven strategies can only enhance the quality of our competitions. Accordingly … we give additional competitive preference to applications that submit moderate evidence of effectiveness that supports their proposed strategies for providing individualized counseling.” The additional preference is worth 2 additional points in an applicant’s score out of a possible 100.

As an illustrative example, such an approach might encourage grant applicants to adopt evidence-based interventions such as InsideTrack College Coaching, a mentoring program for college students that has been shown in a randomized controlled trial to increase the likelihood of college completion by 13 percent over a four-year period compared to the control group. 

Unfortunately, the Department’s modest and sensible step is being strongly opposed by the Council for Opportunity in Education (COE) – the lobbying group that represents organizations delivering interventions that support college attendance and completion by students from poor families. On October 21, the COE President issued an alert attacking the Department of Education for, among other things, reducing the weight given to incumbency of the existing grantees and “emphasizing the importance of evidence-based practices.” Really?  The notice also stated that the Department is “not seeking to partner with colleges and universities nor to recognize the expertise of educators working there.” Subsequently, the COE Vice President of Communications issued a notice for a Capitol Hill briefing entitled “New Policy Changes Threaten Continuation of Federal Program that Helps Low-Income, First Generation Students Complete College.”

Contrary to these misleading claims, the Department is not proposing to cut the program’s funding, let alone threaten its continuation. Rather, its action is designed precisely to improve the program by providing a modest incentive for grantees to adopt interventions with evidence of success. In fact, the purpose is entirely consistent with COE’s own goals of helping disadvantaged youth succeed in college and obtain a degree.

The broader lesson here is that the Administration’s efforts to advance evidence-based program reforms will face challenges from incumbent funding recipients who want program dollars to continue flowing the way they always have – with little regard to evidence about what works. For the evidence-based agenda to succeed, we need to find ways to convince program operators and their lobbyists in Washington that improving their programs based on scientific evidence is in not only in their long-run best interests but can also help ensure continued or even expanded funding. More to the point, it is in the best interests of the students they serve.