SERIES: Brown Center Report on American Education | Number 10 of 15 « Previous | Next »

The 2009 Brown Center Report on American Education: How Well Are American Students Learning?

This year’s Brown Center Report contains studies taking a long view. Part I examines national test data going back to 1971 from the National Assessment of Educational Progress (NAEP). The study in Part II compares the 1989 test scores of more than 1,000 schools to the same schools’ scores in 2009. Part III compares the test scores of conversion charter schools from 1986, when they operated as traditional public schools, to those from 2008, when they operated as charter schools. The studies tackle perennial questions that, as often happens in education, manifest themselves as controversial topics on the contemporary scene: how to interpret trends in test scores, the distribution of achievement, school turnarounds, and charter schools.

Part I rejects the conventional reaction to the 2009 NAEP scores. Scores in fourth-grade math were unchanged from 2007 to 2009. Eighth-grade scores were up a little. Press articles featured expressions of disappointment and concern, primarily from protagonists who used the flat scores to support policy arguments. Part I places the 2009 scores in the context of the 19-year history of the main NAEP, and after comparing the latest scores with results from other equally trustworthy tests of U.S. math achievement, concludes that the hand-wringing is unwarranted.

So when is a purported NAEP trend really a trend? Part I continues by examining achievement gaps, not between two racial, ethnic, or socioeconomic groups, but between the nation’s highest- and lowest-achieving students. It focuses on the distribution of academic achievement instead of the direction of average achievement. The study is a follow-up to a 2009 Fordham Institute paper documenting that the gap between high- and low-achieving students has been shrinking in recent years. The data in Part I show that the trend, which began sometime around 1998 or 1999, is historically unprecedented and extends across subjects (reading and math), grades (fourth and eighth), and tests (long-term trend and main NAEP). It is also more pronounced in public schools than in private schools. The two analyses in Part I highlight the contrast between a trend indicated by data collected from several independent sources over an extended period of time and speculative assertions arising from “instant analysis” of a single set of test scores.

Part II asks a simple question: do schools ever change? The sample consists of 1,156 schools in California that offered an eighth grade in 1989 and 2009. Test scores from 1989 are compared to scores from 2009. The scores are remarkably stable. Of schools in the bottom quartile in 1989—the state’s lowest performers—nearly two-thirds (63.4 percent) scored in the bottom quartile again in 2009. The odds of a bottom quartile school’s rising to the top quartile were about one in seventy (1.4 percent). The reverse was true as well, with similar percentages of top quartile schools staying among the top performers (63.0 percent) or falling to the bottom quartile (2.4 percent). Changes in a school’s socioeconomic status had only a marginal statistical relationship with test score changes.

The persistence of test scores has major implications for today’s push to turn around failing schools. It can be done, but the odds are daunting. California certainly cannot be accused of inactivity in education reform from 1989 to 2009. Few states tried as many diverse, ambitious reforms that targeted every aspect of the school system—finance, governance, curriculum, instruction, and assessment. Not only have these efforts failed to elevate California from its low national ranking on key performance measures, but they have also had little effect on the relative ranking of schools within the state.

The study suggests that people who say we know how to make failing schools into successful ones but merely lack the will to do so are selling snake oil. In fact, successful turnaround stories are marked by idiosyncratic circumstances. The science of turnarounds is weak and devoid of practical, effective strategies for educators to employ. Examples of largescale, system-wide turnarounds are nonexistent. A lot of work needs to be done before the odds of turning around failing schools begin to tip in a favorable direction.

Part III looks at charter schools. Conversion charters are favored by the Obama administration as a restructuring strategy. Most charter schools are start-ups, begun from scratch by their founders. Conversion charters are schools that are traditional public schools and convert to charter school status. They typically continue to rely on their home districts for several functions (e.g., maintenance of buildings, managing pension obligations, transportation services) but are freed from regulations pertaining to curriculum and instruction. The idea is that schools can be more productive if they are allowed to tailor core educational operations to the needs of their students.

California has the largest number of conversions, and the study was able to collect data on two cohorts: 49 schools from 2004 and 60 schools from 2008. For both cohorts, test score data were also available from 1986, allowing a comparison of scores before and after the schools converted. The analysis is exploratory and mainly descriptive. No causal conclusions can be derived from the data.

What do we know about conversions? Test scores look similar before and after conversion. The 2004 cohort evidences a 2 to 3 percentile point advantage as charters, but the 2008 cohort’s scores declined slightly, less than 2 points, from 1986 to 2008. On several key characteristics, conversions look more like traditional public schools than start-up charters. Compared with start-ups, conversions are more concentrated in urban areas, have larger student enrollments, and serve greater numbers of Hispanic and black students. Teachers at conversions are more experienced and more likely to hold teaching certificates, particularly in bilingual education. It is clear that future evaluations of charter schools must differentiate between start-ups and conversions because of the significant institutional differences between the two types of charters.

To sum up, the studies in this year’s Brown Center Report focus on long-term changes. Part I analyzes NAEP data. Parts II and III examine California test scores from the 1980s and compare them to scores from recent years. Because of its long history of testing, California is currently one of the few states able to provide assessment data for such long-term comparisons. That will change as other states continue to test students annually. Creating rich archives of student performance data bodes well for school reform. Improving schools requires patience and persistence, what education professors Richard Elmore and Milbrey McLaughlin1 call “steady work.” It also requires good information to verify whether reforms have paid off, or, like many efforts in education, produced hopeful signs that soon vanish. The future looks bright if analysts’ capacity to peer into the past continues to improve.

SERIES: Brown Center Report on American Education | Number 10