Did you hear about the Head Start evaluation?
Head Start is the iconic federal preschool program intended to enhance the health and school readiness of America’s poor children. Begun as a pilot summer program in 1965 as part of Lyndon Johnson’s War on Poverty, it is now budgeted at $7.235 billion annually for programs serving approximately 1 million children.
The children in Head Start are overwhelmingly poor and minority. They are at high risk of starting school far behind their more advantaged peers, and falling further behind over time. They tune-out and drop-out at alarming rates. In a world in which nearly everything we value, from a long lifespan to financial wealth to family stability, is associated with educational attainment, these children’s lives are in danger. They desperately need a good education. Further, the nation needs for them to succeed in school if it is to achieve its goal of social equity and if it is to complete internationally in a knowledge-based economy. Preschool is where it has to start.
There have long been questions about the effectiveness of Head Start. Supporters of the program traditionally point to decades-old research demonstrating long-term positive outcomes for children from participation in two intensive, small-scale early childhood programs. These programs, Perry Preschool and Abecedarian, suggest the potential of preschool intervention, but they are not Head Start or anything close to it. More recent studies of Head Start itself have suffered from a variety of methodological limitations that limit confidence in their findings.
Enter the Head Start Impact Study. Conceptualized during the Clinton administration by an external advisory committee on which I served, it was to be the best that evaluation science could offer to determine the degree to which Head Start worked, for whom, at what points in schooling, and for what outcomes. To the credit of the Clinton and Bush administrations and the professionals overseeing the study in the Department of Health and Human Services (HHS), the study went forward in 2002 on pretty much the same terms recommended by the advisory committee. It included a nationally representative sample of Head Start programs that had more children seeking admission than available slots. Admission into those programs was determined by lottery. Children who won and lost the lottery were to be followed through the end of third grade, and assessed on a wide variety of outcomes across the domains of cognition, socio-emotional development, health, and parenting. The Head Start Impact Study as designed and implemented is among the strongest evaluations of a government social program ever carried out.
HHS released the Head Start Impact Study Final Report last week. There are several remarkable things about it:
- The study demonstrated that children’s attendance in Head Start has no demonstrable impact on their academic, socio-emotional, or health status at the end of first grade. That’s right. If you were a mother who lost the lottery, couldn’t get your child into Head Start, and had to care for her at home, she was no worse off at the end of first grade than she would have been had she gotten into Head Start. That isn’t to say that she was well off. In the critical area of vocabulary, 3-year-olds entered the study at the 29th percentile in terms of national norms and finished first grade at the 24th percentile whether or not they attended Head Start. That is not good.
- The study went virtually unnoticed. You can’t find anything about it in the Washington Post or the New York Times or the Wall Street Journal or any other media outlet that serves the general public. The Post has 11 reporters covering education. Why isn’t a report on the effectiveness of the nation’s largest federally administered education program, one that serves thousands of needy children within the Post’s metro area, deemed worthy of newsprint? Is Head Start so sacrosanct that bad news about it is to be ignored?
- The report of the study was inexcusably delayed. Data collection for the first grade follow-up was completed in the spring of 2006. Best practice in federal agencies would have seen a report released 12-18 months later. In fact, a draft report was provided to government officials in 2008 but wheels turned for long periods afterwards as the contractors were pushed to try different analytic techniques in the hope that something positive for Head Start could be found. Residuals of that effort are apparent in the released report, wherein findings are reported as suggestive or moderate that do not meet well accepted standards for statistical significance. The inexcusable delays continue as a report on a follow-up at the end of third grade, on which data collection ended in the spring of 2008, is no where in sight.
The federal government through the Office of Management and Budget has, over the years and through different administrations, promulgated a series of directives for federal statistical agencies that go a long way towards protecting the integrity of their products. As a result the public does not have to worry that a government report on number of new unemployment claims has been delayed or subjected to unusual statistical procedures in order to be please those in power. In contrast, no such general protections exist for evaluation studies. The public has no ready way of knowing what evaluations are under way and when reports are expected. The evaluations are often overseen by the same agencies that are responsible for implementing the programs, as was the case with the Head Start evaluation. This is a structural conflict of interest that often leads to pressures on contractors who carry out the evaluations to please their federal clients by casting findings in a more positive light than they deserve. And there is typically no independent peer review of evaluation reports to serve as a check on the integrity of the analysis and reporting of results.
President Obama has called for doing what works in education, health care, and other fields. His administration includes individuals at the highest levels who understand that doing what works depends on timely information from rigorous evaluations, analyzed and reported with independence and integrity. It is time for an executive order or an OMB directive to foster and protect the integrity of program evaluations.
Although the Head Start evaluation report includes findings derived from relaxed statistical standards and was badly delayed, the quality of the study shines through and the message is clear: Head Start isn’t doing the job the families it serves and the nation need. It must be improved. There are many proposals for doing so. Let’s try them, test them, and do what works.