Faith in Common Standards Not Enough

In a recent Brown Center Letter on Education, Don’t Forget Curriculum, we concluded that the effects of curriculum on student achievement are larger, more certain, and less expensive than the effects of popular reforms such as common standards, charter schools, and reconstituting the teacher workforce. We recommended that curriculum have a prominent place in the education reform agenda.

Our findings on the relationship between the quality of state content standards and student academic achievement surprised many readers. To summarize, we found no association between state scores on the National Assessment of Educational Progress (NAEP) and ratings of the quality of state standards. We also drew attention to a little noticed finding from a recent federal study: weak to no association between state performance on NAEP and the stringency of performance standards for state assessments. In other words, when indexed by NAEP scores, differences among states in academic achievement do not seem to be related to differences in the quality of state content standards or the difficulty of passing the state assessment.

The content standards faithful have tried to explain away these findings by suggesting, among other things, that the knowledge tested on NAEP is not sufficiently aligned with the content standards of many states for NAEP to be a valid measure of achievement against those standards. Since there is presently no state-by-state analysis of the alignment of state and NAEP standards we cannot test this hypothesis directly.

We approached the alignment hypothesis indirectly using international data from the Trends in International Math and Science Study (TIMSS). TIMSS, an assessment much like NAEP, is given every few years to representative samples of students from a large number of countries. One would presume that countries whose content standards are best aligned with the content tested by TIMSS would perform better on the assessment.

We drew data on the number of 8th grade TIMSS math topics represented in each of 21 county’s 8th grade content standards from Cogan et al. (2000). We correlated these alignment scores with the average 8th grade TIMSS score for each country in 1995 and 2003, and with gains in those TIMSS scores for each country between 1995 and 2003.

We found a negative relationship between the number of TIMSS topics included in the country’s standards and TIMSS scores for 1995 (r = -.413) and 2003 (r = -.249). The relationship turns positive when analyzing gains in TIMSS scores from 1995 to 2003 but the correlation is small (r = .21) and not statistically significant.

Our analysis suggests that the alignment of content standards and assessments does not play a meaningful role in student performance on TIMSS. Japan, for example, is a consistently high performer on TIMSS, but has fewer TIMSS topics in its 8th grade content standards than any other country in our sample. Other countries, such as Iran, include all of the TIMSS content standards in their curriculum but have low TIMSS scores.

We do not conclude from the present analyses, and those in our previous report, that the quality of standards, the stringency of tests, and the alignment between standards and assessments are unimportant. We can easily imagine that the effects of standards, cut scores, test alignment, and other facets of standards and accountability play out through a set of conditional relationships and interactions. Thus, high-quality common standards may affect student achievement only in a system in which there are also aligned assessments, and aligned curriculum, and accountability for educators, and accountability for students, and aligned professional development, and managerial autonomy for school leaders, and teachers who drawn from the best and brightest, and so on. This is the leaky bucket explanation of our findings, which may be preferred to the hypothesis that standards are no bucket at all. Evidence presently available does not distinguish between the two.

Secretary of Education Arne Duncan has characterized the effort to create common state standards as, “absolutely a national challenge, which we must meet together or we will compromise our future.” Our analyses suggest that the creation of common standards will have little impact on our future in and of itself. Common core standards may be a precondition for other reforms. At the very least, we need a plan for next steps at the state and national level once the NGA/CCSSO Common Core State Standards Initiative is completed, and a theory of action by which those steps together will be sufficient to improve instruction and learning. Faith is not enough.