The challenges of curriculum materials as a reform lever

Executive Summary

There is increasing momentum behind the idea that curriculum materials, including textbooks, represent a powerful lever for education reform. As funders are lining up and state leaders are increasing their policy attention on curriculum materials, this report discusses the very real challenges of this effort. The report draws on my experience over the last several years collecting and analyzing textbook adoption data, as well qualitative interviews of school district leaders and teachers. It identifies challenges in three main areas: collecting and analyzing textbook adoption data; encouraging districts to make different adoption decisions; and encouraging teachers to make different use decisions. The report concludes with specific recommendations, which are aimed primarily at state policymakers who seek to use curriculum materials as a policy reform. 

INTRODUCTION

In widely read Brookings reports, Whitehurst in 2009¹ and Chingos and Whitehurst in 2012² wrote about the impact of curriculum and its potentially transformative power as a lever for reform. Their arguments were straightforward. First, citing recent experimental studies, they documented that curriculum materials can have large direct effects on student learning³. Second, they noted that school and district leaders could not make textbook adoption decisions on the basis of textbook quality, because such evidence did not widely exist. Third, they claimed that there was little data of even a descriptive nature on textbook adoption patterns and practices, but that this would be relatively easy to collect. And finally, they argued that if the above issues were solved, textbooks could be an inexpensive (both politically and in dollars and cents) reform strategy.

The Chingos and Whitehurst report ended with specific recommendations, including the following: a) State education agencies should collect data from districts on the instructional materials in use in their schools; b) the NGA and CCSSO should put their weight behind the effort to improve the collection of information on instructional materials; and c) foundations could provide the start-up funding needed to collect data on instructional materials and support the research that would put those data to use.

While it is of course not known whether their report is the direct impetus, it is clear that some of the recommendations they made are coming to fruition. For example, the Gates Foundation is moving into the area of curriculum materials,⁴ and other funders appear interested as well. Chiefs for Change recently released a statement on the importance of curriculum materials and the role of state departments of education in collecting better data on the topic.⁵ And a number of researchers, myself included, have begun paying attention to curriculum as a reform lever and collecting and using data to analyze the impact of materials on student achievement.⁶

But how good are the prospects for this as a serious reform effort? And what are the potential barriers? The purpose of this report is to take stock of where we are and to offer suggestions for this effort moving forward.⁷ To answer these questions, I draw on three main sources. One source is my recent efforts to collect textbook adoption data in the five largest U.S. states – I draw on my experiences in doing this work and also the data that we have ultimately collected and analyzed.⁸ A second source is a set of interviews of school district leaders in the state of California focusing on their districts’ textbook adoption policies and practices. And a third source is evidence on teachers’ use of textbooks, drawn from both my data collection efforts through interviews and others’ efforts through surveys.

In what follows, I organize my discussion around what I view as the three main areas of challenge:

• The challenge of collecting and analyzing textbook adoption data to determine which books are most effective.
• The challenge of getting the most effective books in teachers’ hands (i.e., through school and district textbook adoptions).
• The challenge of getting teachers to use these books once they have them.

I conclude with specific recommendations for how to overcome these challenges. While I am optimistic about this reform strategy, textbooks will not be a successful reform without serious, sustained engagement along these dimensions. 

The challenges of collecting and analyzing textbook adoption data 

My team’s experiences in collecting and analyzing textbook adoption data suggest that there will be a number of hurdles if states seek to undertake this kind of effort.

Collecting the Data

Textbook titles seem like straightforward pieces of data to collect, but in fact the issue is more complicated than it may seem. Even well resourced state departments of education may struggle to simply collect the data in ways that make it usable for the kinds of research Chingos & Whitehurst recommend.

First, there is the simple fact that even a piece of information as seemingly innocuous as textbook titles may be seen as having political implications. And these implications may lead to resistance to sharing the data. For instance, teachers or district leaders may worry that collecting data on textbook adoptions is the camel’s nose under the tent that may lead to more prescriptive state control over curriculum issues (which are historically the bailiwick of local authorities). Unless the collection is made mandatory and routine, then, there will likely be some resistance to sharing the data. But the more prescriptive the effort is, the more educators’ hackles may be raised.

Second, there is the complication of whom to ask for the data. Setting aside schools of choice, districts are very likely the units responsible for making the actual purchases in most or all states. But in some states, districts are typically “uniform adopters” (all schools in the district use each adopted book), and in other states they are not. Will states really get in the business of surveying every school in a state to gather this information? If they survey districts, will districts actually know what books are used in schools? Respondents—either district or school—may also lack key information such as adoption years, which are necessary for the most sophisticated analytic approaches.

Third, there are many complications in identifying books that make this task more difficult than it may seem at first blush. Many book series have multiple editions—Pearson’s enVision Math had state-specific versions, then a Common Core version, and now an enVision 2.0 version—it is easy to confuse these in data entry. Districts/schools may differ in the type of license they select—digital materials, consumable books, multi-year licenses, etc.—will this kind of information be collected? Some of these problems could be solved by collecting ISBN numbers, as suggested by Whitehurst and Chingos, but there are challenges with that approach as well (people may be less willing to fill out surveys if they have to go find ISBN numbers, for instance). Then of course schools and districts have up to 13 grades, multiple subjects, and sometimes four or more academic tracks—will textbook information be collected on all of these, or just some? How will titles be linked to courses?

Fourth, while some of the complications just mentioned could be avoided if data were collected each time materials were purchased, many districts do not purchase books. Some districts use materials like EngageNY—full-year materials that they obtain for free online. Other districts assemble or develop their own materials. How would a state data collection account for these eventualities?

Analyzing the data

Once the data are collected, there are multiple analytic strategies that can be used to determine which textbooks work best, and there is not consensus on the best approach. Some researchers have used matching or other regression-based approaches with school-level achievement data, while others have used student-level data and value-added analyses. Regardless, the goal is to identify the causal effect of districts’ choice of one textbook over another.

In general terms, the main methodological concern is to eliminate selection bias so that the identified “impacts” of a textbook are not actually attributable to some other preexisting difference among districts choosing one textbook over another. Koedel and his coauthors have used various matching approaches, presenting evidence that districts’ textbook adoption choices are not strongly related to observable school and district characteristics.⁹ They also conduct a series of falsification tests that provide convincing evidence that selection is not at play. However, in unpublished analyses, my colleagues and I have investigated textbook effects in other subjects (science), other grades (middle school mathematics), and other time periods (post-Common Core) and found that there seems to be more evidence of selection bias in those areas than in prior studies. Specifically, we have found sometimes large differences among schools adopting particular textbooks in terms of prior achievement or other demographic variables. This could be evidence that something is changing in textbook adoptions to make selection bias more of a concern. Regardless of the specific technical concern, the point is that the science on using observational data to estimate textbook impacts is far from settled, and the methods that work in one instance may not work in all others. Furthermore, the time and resources to get this analytic work done may be substantial.

trying to get schools or districts to make better adoptions

Suppose we were able to collect good enough textbook adoption data from large numbers of schools and districts and use it to calculate impact estimates for each book. Would districts make use of these data in their adoption decisions? My research team’s interviews with school district leaders suggest many reasons why they may not. Of course, it is possible that if the data were better we might have found different things in our interviews, but there are likely some real barriers to getting districts to make different/better adoption decisions.

First, the very decentralized nature of educational governance makes getting virtually any reform adopted at scale a real challenge. In some states (approximately half) the state is involved in school and district textbook adoptions by putting out a formally approved list of materials in certain grades and subjects. California does this, though California’s textbook adoption list is advisory—districts are not required to purchase off the list. In states with these kinds of lists, getting the most effective books to appear on the state list would go a long way toward getting the best books in the most schools. But in California we found around a quarter of schools used books from off the state-approved list. And many other interviewees, even in districts that adopted from the state list, expressed concern about the quality of the state’s review process. In states without a list, schools and districts are generally on their own to figure out which books to adopt.¹⁰ In short, changes to state laws or policies that strengthen the role of the state in textbook adoptions would probably be helpful if the goal was widespread adoption of the most effective books.¹¹

Second, school districts have complex, highly ceremonial practices when it comes to textbook adoptions, which would likely be a barrier to more streamlined forms of decision making. We found in our interviews that virtually all districts have processes that involve a) one or more committees of teachers, b) evaluation of textbooks against complex rubrics (even in the case where the books were on the state list and thus had already been evaluated), c) multi-week pilots, and d) one or more formal votes before reaching a final decision. While better evidence could certainly be fed into this process along the way, it is far from guaranteed that the process would result in the best books being chosen.

Third, the timeline for getting evidence of effectiveness in the hands of district leaders to inform their decisions is challenging at best, and impossible at worst (though this depends at least in part on how states handle revisions to their standards over time). In the core subjects, most states have standards adoption and revision cycles every 7 to 10 years. Publishers put out new versions of books perhaps two years after a new set of standards is adopted, and states put out their lists thereafter (for instance, California put out its math textbook list during the 2013-14 school year, approximately three years after the adoption of Common Core). In order for impact estimates to be calculated, there needs to be a reasonably large number of districts adopting books and using them for at least a couple years. This would mean that the earliest impact estimates could probably have been obtained post-Common Core would have been perhaps 2016. By this point, almost every district in the state had already made an adoption purchase, meaning they were/are not looking to make another purchase soon. By the time the next textbook adoption cycle happens at the district level the standards will have been in place for approximately a decade (assuming they are not dramatically changed in the interim), and it is not even clear that publishers will be publishing the same versions of their books. Of course, if the standards stay stable and the published books stay mostly unchanged, then the results could be useful to districts making another adoption at that time, but this is a large number of contingencies given the transient nature of education policy.

Encouraging teachers to make better textbook use decisions

The fact is that, while many teachers still use textbooks, large proportions of teachers use them as simply one resource among many. This finding is confirmed in both large, state- and nationally-representative surveys, as well as in our interviews of California teachers. In our 67 interviews, no teachers said they used only the district-adopted textbook for their 8^th grade mathematics instruction. Most teachers reported that the adopted book was inadequate in one of two ways—it lacked sufficient opportunity for students to practice foundational skills, or it lacked sufficient enrichment exercises to cover the more conceptual content in the standards. Whatever the gap in the materials, teachers reported supplementing with lessons from old books or with materials they sourced from various websites on the internet. An illustrative quote from one of our teachers was ““We have had to use additional resources. We can’t just settle on just using the [Textbook Title]. There isn’t enough quality in it in order to make it a full, 100 percent program. If you just used the book itself and nothing else, it wouldn’t be enough for them to learn the entire curriculum.” Given this view of textbooks, getting even the best-quality materials to be used with fidelity by teachers may be a challenge.

Teacher surveys suggest that textbooks may not be the main source of lessons for large proportions of teachers. For instance, a five-state study found that 72-80 percent of teachers (depending on subject) reported using instructional materials developed by them or their colleagues at their school at least once a week, as compared to 43-53 percent for materials created by external organizations such as publishers.¹² Another national survey pegged the proportion using district-adopted textbooks once or more a week at about 62 percent.¹³ National data from the American Teacher Panel found greater than 90 percent of teachers reported using Google, and more than 70 percent reported using TeachersPayTeachers and Pinterest, to find lessons.¹⁴ Regardless of the data source, it is clear that textbooks are widely used but are far from the only source of curriculum in typical American classrooms. Furthermore, these numbers are quite a bit lower than those cited in Chingos and Whitehurst’s report,¹⁵ suggesting that the use of textbooks has declined over time. Certainly it is possible that this could change if teachers had better books available, but textbook reform would likely affect a modest proportion of the curriculum of the typical classroom.

To be sure, our teacher interviews did find certain district-level policies that seemed to be associated with better implementation of standards. For example, we found that teachers did need some sort of backbone for their curriculum, and having a formal textbook adoption provided that. Teachers in districts that did not formally adopt a curriculum, or that took a very long time after the standards were written to do so, complained about the lack of support and their concomitant inability to fully implement the standards. So districts should adopt something, and it’s possible that a stronger backbone—offered by a more effective textbook—would be used even more. In addition, teachers said they needed specific kinds of professional development focused on both the textbook itself and the standards more generally. They were critical of publisher-provided professional development, which they said often focused on surface elements of the materials. And they often were unable to state specific changes that were called for by the standards, perhaps reflecting a lack of deep knowledge of the standards. In short, teachers in general would like to have both a formally adopted material and support to understand and implement the standards through professional learning opportunities.

recommendations

There are good reasons to believe that curriculum materials can serve as an important reform lever. But this report has laid out some of the challenges in getting this reform to achieve its desired impact. Based on these issues, I make the following recommendations.

In terms of data and analysis:

1. The best approach will be to routinize data collection, perhaps at the time of purchase, for each district in the state. If this is not possible, embedding annual data collections in other data collection activities could also work, but recalling will always have more error—and probably more burden—than more automated approaches.
2. The state should decide which subjects, grades, and courses will be the target of its collection efforts. This decision might be informed by surveying educators to understand where textbooks are currently most used.
3. At a minimum, the state should collect titles/publishers, editions, and adoption years for any book on which they gather information. Again if this is done routinely at purchase it would be straightforward.
4. The state should consider what it wants to collect from districts or schools that do not claim to use any formal textbook. Short surveys or audits of curriculum materials from samples of teachers in those sites may be the best approach. It would not be appropriate to collect no data simply because the district does not use textbooks—access to quality curriculum is an equity issue that is under the state’s purview.
5. Rather than merely collecting the data and hoping someone analyzes them, the state should have in place plans or a relationship that ensures the data get routinely analyzed by qualified researchers or staff. Otherwise this is unlikely to happen.

While these recommendations will not ensure that trustworthy impact estimates will be created, they will go a long way toward ensuring that the conditions at least exist.

In terms of district adoption and teacher use:

1. There appears to be little reason for states not to put out lists of quality materials. These lists can drive adoption decisions and can simplify the task of adopting for schools and districts.¹⁶ That said, states should be sure that their adoption processes are transparent and high quality so educators can trust the results. In states where such a move would be politically feasible, they should consider incentivizing or mandating districts purchase off the state-approved list.
2. Intermediary organizations, like California’s County Offices or New York’s BOCES, can serve an important role in helping smaller districts collaborate on, select, and implement materials. States should consider supporting these organizations directly for this purpose.
3. Because the use of non-textbook resources is large and growing, states should consider evaluating these for quality and creating curated lists of approved supplementary resources. They could also work with districts or intermediary organizations for this effort. Doing this might ease the curriculum selection burden for teachers and result in better quality materials in teachers’ hands.
4. Similarly, states should consider getting in the business of supporting quality professional development aligned with the standards and to support the implementation of the top-rated curriculum materials. This could ease the burden on schools and districts and prevent them from having to find or create their own learning opportunities.
5. Finally, states should plan regular data collection and analysis related to teacher adoption and use of curriculum materials. They might specifically work with districts that are adopting new materials to use those opportunities to research implementation and effects.

Together with the data collection and analysis activities described above, these efforts are likely to help ensure public school students have access to high quality curriculum in all of the state’s schools. Without these kinds of sustained efforts, the strategy of using curriculum materials to leverage educational improvements may be unlikely to succeed in the long-term.

The author did not receive any financial support from any firm or person for this article or from any firm or person with a financial or political interest in this article. He is currently not an officer, director, or board member of any organization with an interest in this article.