The Harlem Children’s Zone Revisited

Our recent report on the effectiveness of the Harlem Children’s Zone (HCZ) relative to other charter schools in New York City generated a public response from Geoffrey Canada, the CEO of HCZ.

Our issue is not with the HCZ as a philanthropically supported endeavor to improve the lives of children in Harlem, but with the use of the HCZ as evidence that investments in wraparound support services and neighborhood improvements are a cost effective approach to increasing academic achievement. In an era of stress on public budgets, we think there should be good evidence that an expensive new approach works before it is scaled up and widely implemented with taxpayer funds. Our findings and our view are that the HCZ does not provide that evidence. Our quarrel is not with the HCZ but with the evidence for the Obama administration’s request to Congress for $210 million to replicate the HCZ in 20 communities across the nation.

Several of Canada’s statements cannot be addressed with data and we will respond to those in turn. But first we address the two issues he raises that allow an empirical test.

Empirically addressable issues

The first is that we “arbitrarily tossed out the achievements of… Promise Academy II charter school.” Our decision was not in our view arbitrary. We wanted to capture as many data points as possible from the three years of achievement data in the New York State database and did not want the HCZ to have to compete against itself in our ranking procedure. That there were 20 data points for Promise Academy but only 6 for Academy II, and that students had been enrolled in Promise Academy longer than Academy II argued for our approach. Nevertheless we take Canada’s claim seriously, which is that leaving out Academy II distorts the results.

The second empirically testable claim is that our report “used incorrect demographic data about HCZ’s charter schools.” Canada indicates that many parents of students in the HCZ charter schools who are eligible for the federal free and reduced price lunch program do not fill out the required federal forms, and that the actually number of eligible students is “over 80 percent” rather than the “33 to 52 percent” in the public records (the actual numbers in the state database for the HCZ Promise Academy are 33 percent to 65 percent).

The problem of school statistics on free and reduced price lunch eligibility underestimating the extent of poverty in the families served by schools is well known and hardly unique to the HCZ. Thus the state statistics for the other charter schools in our sample are likely to be on the low side as well. Further, we were criticized for using incorrect data, when what Canada identifies as the correct data in his press release are found in no public source of which we are aware..

The following table reports the results from a new analysis we have conducted that incorporates data from the HCZ Academy II and uses either 80% for free and reduced priced lunch eligibility (Canada’s number) or the state reported number if it is greater than 80 percent as the HCZ entry for all comparisons. Our previous analysis entered free lunch eligibility and reduced price eligibility as separate predictors since they are reported separately in the state database. To incorporate Canada’s number, which combines the two categories, we collapsed the two categories into one for all schools in our new analysis. We incorporate the HCZ Academy II in our new analysis by averaging its results with those of Promise Academy for the grades and years in which both schools have data, weighted by the number of students taking the state assessment in each school. To ease comparison with our previous report, we include those findings as well in the table below. As in the previous report the table presents the percentile ranks for HCZ relative to other charter schools in Manhattan and the Bronx.  

 As casual inspection of the table makes clear, very little changes as a result of analyzing the data using Canada’s figure for free and reduced price lunch eligibility and including HCZ Academy II. For both raw achievement scores and scores adjusted for school demographics, students attending HCZ schools are in the middle of the distribution of charter schools in Manhattan and the Bronx.

Other Issues

1) We incorrectly state that school students who live outside the Zone do not receive the full complement of HCZ services.

It is not our statement that access to wraparound services and neighborhood supports is greater for families who live in or close to the Zone than for families who live outside the Zone. We simply described the research of Dobbie and Fyrer, which is prominently featured on the HCZ website.

2) We give equal weight to each grade without looking at the progress each cohort has made over their time.

The state database on student achievement does not include data on student progress. It includes only the results at the end of each school year for students who took the state test in that year. Because students move in and out of charter schools, are present or absent on the day of testing, and take different tests in each grade, the end of the year results for math for 8th graders in 2009 cannot be compared with the end of the year results for math for 6th graders in 2007 as a measure of cohort progress. We applied the same ruler to all charter schools, which equally weights each grade and subject. Students improve with time in other charter schools as well. Finally, the bar graph in our paper presented results for math at 8th grade in 2009, which is where the HCZ should look best if its students improve over time. We intended to present the HCZ graphically in as favorable a light as possible. The HCZ is in the middle of the pack of charter schools in math at 8th grade in 2009, just as it is when its results are averaged over all grades and subjects.

3) We trivialize the HCZ’s accomplishments.

Our report was about the HCZ as an education reform. On that score we wrote that the “HCZ works” to raise student achievement and that “students attending the HCZ Promise Academy are doing impressively better than students of their backgrounds attending a typical public school in NYC.” Not only did we credit the effectiveness of the HCZ charter schools in our previous report, we here applaud Geoffrey Canada’s life work, which is to give poor minority children in Harlem a set of life experiences more like those that are available to middle class children in this country with the hope that this will fundamentally improve the trajectory of their lives.

Rather than trivializing the HCZ, we would like to see it continue, thrive, and be the subject of evaluations that will address its impact in a more thorough and long-term way than can be accomplished with the data currently available.