What would you nominate as an effective government program? Enter our contest! Winner to receive a copy of Ron Haskins’ new book, Show Me the Evidence. Enter your nominee in the blog comments below.
As I argue in my new book, Show Me the Evidence (co-authored with Greg Margolis), the last six years have seen the most impressive expansion of evidence-based policy in the history of federal social programs. The Obama administration has launched a series of evidence-based initiatives that have the potential to revolutionize the way the federal government funds social programs and what program sponsors at the state and local level must do to win and retain federal dollars. Specifically, grantees must show they are spending their federal dollars on programs that have evidence from rigorous evaluations of producing positive impacts on children’s development or achievement as measured by outcomes such as teen pregnancy, educational achievement or graduation rates, performance at community colleges, employment and earnings as young adults, or reducing rates of incarceration. Second, they must evaluate their programs using scientific designs to ensure that they are continuing to have impacts and to reform the programs if they are not.
This strategy requires a pipeline of social programs that have been tested and shown to be effective by rigorous evaluations. However, experience shows that most social programs, including some of the most celebrated such as DARE and Head Start, produce modest or no impacts that last when subjected to rigorous evaluations. An important virtue of focusing on evidence is not simply that the public will have reliable information about whether programs work, but that the evidence places pressure on programs to change and improve when they are not working.
In addition to improving existing programs, government and foundation research dollars are being used to develop and obtain evidence on new programs, some of which have been found to produce significant benefits for children or adolescents by rigorous evaluations. The most important claim of the Obama evidence-based initiatives is that if government spends its intervention dollars on programs shown by rigorous evaluations to work while simultaneously using rigorous evaluations to improve existing programs and develop new programs that work, in the long run federal spending on social programs can produce more benefits for more children and move the needle on the nation’s most important social programs.
But do we really have examples of social programs that produce these hefty impacts on social problems? The answer is a resounding yes. What follows are overviews of five of my favorite programs, all of which have produced big and lasting impacts on social problems (most of the overviews were adapted from the website of the Coalition for Evidence-Based Policy; the summary of the Small Schools of Choice program was taken from the website of MDRC, a prominent program evaluation firm):
Career Academies are high school education programs that have three distinguishing characteristics:
- They are organized as small learning communities (150 to 200 students) to create a more supportive, personalized learning environment;
- They combine academic with career and technical curricula around a career theme; and
- They establish partnerships with local employers to provide career awareness and work-based learning opportunities for students.
Each Academy typically focuses on a specific field (e.g., health care). Students enter a Career Academy in 9th or 10th grade, and are taught by a single team of teachers through grade 12. The most powerful evidence of the impact of Career Academies is provided by a large, multi-site, randomized controlled trial. The trial evaluated nine Career Academies in high schools located in or near large urban school districts across the United States. These Academies had each implemented and sustained the core features of the Academy model for at least two years. They represented a variety of the career themes that Academies typically offer (e.g., technical, service-oriented, or business-related). The effects summarized here were obtained 8 years after the students’ scheduled high school graduation:
- 11% increase in average annual earnings – i.e., $2,460 per year – over the previous eight years ($24,560 in annual earnings for the Career Academy group versus $22,100 for the control group).
- The earnings effect was sustained over the full eight years, and showed no sign of diminishing.
- The earnings effect was concentrated among men, who experienced a 17% increase in annual earnings over the follow-up period. There was no statistically significant effect on women’s earnings.
- 23% increase in the likelihood of living with a child and partner.
- 35% decrease in the likelihood of being a non-custodial parent (5% for the Career Academy group versus 8% for the control group).
- The approximate 3-year cost of $2,300 per student was at least partly (and perhaps fully) offset by the increased tax revenue resulting from the gain in earnings of Career Academy students and perhaps by reduced use of social programs as well.
The Nurse-Family Partnership (NFP) program provides nurse home visits to pregnant women with no previous live births, most of whom are low-income, unmarried, and teenagers. The nurses visit the mothers approximately once per month during pregnancy and the first two years of their children’s lives. The nurses teach positive health related behaviors, competent care of children, and maternal personal development (family planning, educational achievement, and participation in the workforce). The program costs approximately $13,600 per woman over the three years of visits.
The evidence supporting NFP is contained in three randomized controlled trials (RCTs) of the program (see here, here, and here). The three trials – each carried out in a different population and setting – all found the program to produce sizable, sustained effects on important mother and child outcomes. The replications of the NFP intervention program in multiple sites (in New York, Tennessee, and Denver) provide confidence that the program would be effective if faithfully replicated with other, similar populations and settings. However, the specific types of effects often differed across the three studies. The specific effects that were replicated, with no countervailing findings, in two or more of the trials – and thus are the most likely to be reproducible in a program replication – are:
- reduction in measures of child abuse and neglect (including injuries and accidents);
- reduction in the number of subsequent births during the mothers’ late teens and early twenties;
- reduction in prenatal smoking among mothers who smoked at the start of the study; and
- improvement in cognitive and/or academic outcomes for children born to mothers with low psychological resources (i.e., low intelligence, mental health problems, lack of self-confidence).
Of special note because of its long-term follow-up is the original RCT, conducted in Elmira, NY, beginning in the late 1970s. Women were randomly assigned either to a group given the opportunity to participate in the Nurse-Family Partnership, or a control group that was provided developmental screening and referral to treatment for their child at ages 1 and 2 and, in some cases, free transportation to prenatal and well-child care. Approximately 90% of the women were white, 60% were low income, and 60% were unmarried. Their average age was 19. Here is an overview of the results for the Elmira trial at the final follow-up study after 15 years:
- 48% fewer officially-verified incidents of child abuse and neglect as of age 15 (an average of 0.26 incidents per nurse-visited child versus 0.50 per control-group child).
- 43% less likely to have been arrested, and 58% less likely to have been convicted, as of age 19 (21% of nurse-visited children had been arrested versus 37% of control-group children, and 12% versus 28% had been convicted, according to self-reports).
- 57% fewer lifetime arrests and 66% fewer lifetime convictions (an average of 0.37 versus 0.86 arrests, and 0.20 versus 0.58 convictions, according to self reports).
- 20% less time spent on welfare (an average of 53 months per nurse-visited woman versus 66 months per woman in the control group).
- 19% fewer subsequent births (an average of 1.3 births versus 1.6).
- 61% fewer self-reported arrests (an average of 0.13 versus 0.33).
- 72% fewer self-reported convictions (an average of 0.05 versus 0.18).
The cost of three years of home visits by a trained nurse using the Nurse-Family Partnership model is $13,600. There are numerous outcomes found in one or more of the trials that save government spending. These include a 20–50 percent reduction in child abuse and neglect, a 10-20 percent reduction in subsequent births in the late teens and early 20s, and reduced welfare payments.
Carrera Adolescent Pregnancy Prevention Program
Sponsored by the Children’s Aid Society, the Carrera Adolescent Pregnancy Prevention program is a comprehensive youth development program for economically disadvantaged teens who enter the program at ages 13-15 and usually participate for three years. The program is provided after school at local community centers, and runs for about three hours each weekday after school.1 The program includes five main activities:
- Daily academic assistance (e.g., tutoring, homework help, assistance with college applications);
- Job Club 1-2 times per week, including such activities as learning to complete a job application and interview for a job;
- Family life and sex education 1-2 times per week, led by a reproductive health counselor;
- Arts activities 1-2 times per week (e.g. music, dance, writing, or drama workshops); and
- Individual sports activities 1-2 times per week (e.g. tennis, swimming, martial arts).
The program also provides free mental health and medical care through alliances with local health care providers. A key component is reproductive health care, including physical exams, testing for sexually transmitted infections, a range of contraceptive options, and counseling. Carrera program staff schedule the teens’ health appointments and accompany them on their visits. The program costs approximately $4,750 per teen per year to implement (2009 dollars).
The evidence to support the program comes from a large, multi-site RCT (linked above). This trial evaluated the program as implemented in 12 well-managed community youth agencies in 6 states during the period 1997-2004; 1,163 teens aged 13-15, who were not parenting or pregnant, participated in the evaluation; 45% of the teens were African American or Caribbean black, and 29% were Hispanic. 58% were from single or no-parent households, and 54% lived in households that had no employed adult and/or received entitlement benefits (e.g., public assistance, Medicaid).
On average, Carrera group teens attended program activities for 12 hours per month during the three years after random assignment. At the end of the third year, 70% of the Carrera group teens were still involved in the program. Here is overview of the program impacts at the end of the program:
- For Carrera group females:
- 40% less likely to have ever been pregnant (15% of Carrera group females had been pregnant vs. 25% of control group females).
- 50% less likely to have ever given birth (5% vs. 10%).
- More than twice as likely to be using Depo-Provera — a highly effective hormonal contraceptive — at last intercourse (22% vs. 9%).
- For Carrera group males, there were not significant effects.
- For the full sample (males + females):
- 7% reduction in likelihood of having had teen sex (statistically significant at the 0.10 level but not the 0.05 level).
- 16% more likely to have had some work experience (89% of the Carrera group vs. 77% of the control group).
- Positive effects on some educational outcomes (PSAT scores and college visits) but not others (e.g. grades).
- Effects at 7 years after random assignment, at average age 21:
- 30% more likely to have graduated high school or obtained a G.E.D. (86% of the Carrera group had graduated or obtained G.E.D. vs. 66% of the control group).
- 37% more likely to be enrolled in college (63% vs. 46%).
The Carrera program produced many outcomes that reduce government spending. These include substantial reductions in teen pregnancy rates, a 30 percent increase in high school graduation or GED achievement, and a 37 percent increase in college enrollment. Just the increased enrollment in college is likely to offset all the program costs in the long run.
Success For All
Success for All is a comprehensive school-wide reform program, primarily for high-poverty elementary schools, with a strong emphasis on early detection and prevention of reading problems before they become serious. Key program elements include daily 90-minute reading classes, each of which is formed by grouping together students of various ages who read at the same performance level; a K-1 reading curriculum that focuses on language development (e.g., reading stories to students and having them re-tell), teaching students the distinct sounds that make up words (i.e. phonemic awareness), blending sounds to form words, and developing reading fluency; daily one-on-one tutoring (in addition to regular classes) for students needing extra help with reading; and cooperative learning activities (in which students work together in teams or pairs) starting in the grade 2 reading classes.
The evidence of Success for All’s effectiveness is based a research design in which 41 schools across 11 states were randomly assigned to an experimental or control group. Grades K-2 but not grades 3-5 were included in the evaluation Prior to random assignment, at least 80% of the schools’ teachers had voted in favor of adopting Success for All and the schools had agreed to allow data collection over the course of the study. The schools contained a total of 2,694 entering kindergarten students administered a pretest at the start of the study. The student population in these 41 schools was 56% African-American and 10% Hispanic, and 72% of students were low-income (i.e., eligible for federally subsidized lunches). Approximately three years after random assignment, the study assessed reading outcomes for all 2nd-grade students in the sample schools. Sixty-nine percent of these students had been exposed to Success for All, or the control condition, for all three years of the study (i.e., in grades K-2); the other 31% had enrolled in the Success for All or control schools during the study, and so had received partial exposure.
Here is an overview of the effects of Success for All on school-wide second-grade reading outcomes, three years after random assignment (versus the control schools):
- On average, 2nd-graders at Success for All schools –
- Passage comprehension: From an effect size of -0.10 in year 1, to 0.12 in year 2, to 0.21 in year 3.
- Word identification skills: From 0.09 to 0.19 to 0.24.
- Word attack skills: From 0.32 to 0.29 to 0.36.
- Scored higher in passage reading comprehension than approximately 58% of their counterparts at control group schools (this equates to a standardized effect size of 0.21).
- Scored higher in word identification skills than approximately 60% of their counterparts at control group schools (this equates to a standardized effect size of 0.24); and
- Scored higher in word attack skills than approximately 64% of their counterparts at control group schools (this equates to a standardized effect size of 0.36).
- To express these effects as grade level equivalents: On average, 2nd-graders at Success for All schools score approximately 25-30% of a grade level higher in reading ability than their counterparts at the control schools.1
- The program’s effects generally grew in size from the first to the third year of the study:
- This was a large, multi-site study evaluating Success for All as it is typically implemented in high-poverty elementary schools, thus providing evidence about the program’s effectiveness in real-world public school settings.
- The study had a reasonably long-term follow-up, and low-to-moderate attrition: Three years after random assignment, reading test scores were obtained for students in 85% of the sample schools – i.e., 35 of the original 41. (Of the six schools lost at follow-up, five closed due to insufficient enrollment and one dropped the Success for All model due to local political problems and refused to participate in data collection.) The number of schools lost in the Success for All versus control group was the same (3 each).
Success for All costs about $510 per student for the 3-year program. The major impacts shown to date are on school achievement, primarily reading skills. This increase in skills is likely related to high school graduation and college enrollment rates, both of which have been shown to produce reductions in government spending in the long run. Whether Success for All has impacts on long-term measures such as these remains to be seen.
Small Schools of Choice
In 2002 New York City closed 31 large, failing high schools and replaced them with small schools of choice (SSC) that featured specialized curriculums, close associations with outside groups such as businesses and non-profit organizations, and teachers and principals who developed their school philosophy together and advertised it to students and parents. Students entering high school (at grade 9) were allowed to apply to several schools. As a result of student (and parent) self-selection, 105 of the SSCs were oversubscribed. This overflow of students caused the New York school system to randomly assign students to the SSCs and other types of schools. This procedure was repeated for four consecutive years, creating the opportunity to study four cohorts with a total of about 21,000 students who were assigned randomly to either an SSC or a different type of school. Nearly 95% of the students were black or Latino and nearly 85% of the students were from low-income families as measured by eligibility for free or reduced-price school lunches.
As shown in several reports by the research firm MDRC (see here and here), the SSC schools have produced substantial impacts on two measures that have been difficult to impact in previous education evaluations:
- Students in SSC’s had significantly higher graduation rates than control students (71.6% vs. 62.2%).
- Students in SSC’s had significantly higher rates of enrollment in colleges (49.0% vs. 40.6%).
These impacts are achieved despite the finding that “the cost per high school graduate is substantially lower for the small-school enrollees than for their control group counterparts.” This favorable cost result is achieved because although the per-pupil cost of control schools is about the same as SSCs, students at control schools are more likely to require a fifth year of schooling to graduate and they are less likely to graduate at all. Beyond this finding, which shows cost savings for government at the time the program takes place, the increased high school graduation rate and the higher college entry rate will likely produce benefits to the individuals involved, especially in their lifetime earnings, and to government in the form of increased taxes and reduced payments for welfare programs in the future.
There are some structural negative changes that make some people in particular more vulnerable. And in the end, mental health just reflects that. ... For young people who do not have good higher levels of education, what they're going to do in the future is very unknown.