THE SITUATION
Funding research is one of the widely accepted roles of government. Recognizing this, the federal government currently funds, directly and indirectly, $125 billion per year across a wide variety of areas—developing weapons systems for the military, creating new technologies to protect the environment, finding breakthroughs to cure disease, and more. That makes sense, as the knowledge that comes from research is a public good that everyone can use.
Only 1.6 percent of these funds, however, goes toward research about education.[1] This low spending level is ironic considering the widely held beliefs that our education system is underperforming and that human capital is a core investment for future social and economic prosperity. Also, education comprises more than six percent of the national economy, far larger than the sector’s share of research and development spending. This suggests we should be spending far more on education research than we are today.
The underinvestment in education research is especially striking given the new Every Student Succeeds Act (ESSA). The new law returns to the states control over some elements of schooling that the federal government had taken under No Child Left Behind but also reinforces the requirement for evidence-based decisionmaking. Unfortunately, it will be difficult for states to meet these new and improved evidence standards given the current state of the education research enterprise.
To be sure, tremendous progress has been made in research over the last 15 years. As the Government Accounting Office (GAO) has noted, the creation of the Institute of Education Sciences (IES) within the U.S. Department of Education (ED) has instilled a focus on scientific rigor and helped build a cadre of PhD-level researchers trained to carry out this work. By sending the message that rigor matters and expanding the capacity to produce research meeting those rigorous standards, the federal government has created a steady pipeline of informative research.
Ensuring that the research makes an impact is a tougher challenge. In some respects, there is an inherent tension. Researchers focus on rigor and making sure their results are valid for the program and population of students they are studying. That all takes time. But these research considerations are cold comfort to policymakers who have to make decisions now about students, programs, and implementation that differ from those in the available studies. Fortunately, there is a productive middle ground: if researchers regularly reported on implementation and how their results vary across student groups, this would go a long way toward helping policymakers interpret the results. Releasing this more informative research more widely would also help ensure that the latest findings get into the right hands when education decisions are being made. IES has been working hard on this problem as well. The improvements they have made are ones that other federal agencies are starting to follow, but there is still more to do.
Current research also misses the mark because the research questions themselves are typically developed, answered, and communicated mostly by researchers. Evidence-based decisionmaking is only feasible on a large scale if researchers are addressing the questions of policymakers. Given the complexity of schooling and the competing pressures education leaders face, simply creating a great website of research findings is far from enough. School systems are run on relationships, trust, and common understanding—all of which are limited in the American research enterprise. Research will have greater impact if done with policymakers and practitioners. We still have much work to do to make our system a true “laboratory of democracy” and one in which research is a regular feature at the decisionmaking table.
IES, foundations, and others have recognized the challenge of getting to evidence-informed practice, and they have responded by supporting a growing list of research-practice partnerships. But much remains to be done. We need a research architecture to produce more research more quickly—and in partnership with practitioners—while still meeting high standards for rigor. While this is no easy task, I argue below that numerous opportunities exist to make the leap to true evidence-informed policy.
RECOMMENDATIONS
To ensure that research is at the table when important education decisions are made, we need three major shifts in the federal role in education research:
- Increase the capacity of state and local education agencies to carry out and interpret research themselves so they can build a culture of evaluation and evidence and learn from their own daily experiences.
- Improve the speed and usefulness of research produced by external researchers by making more and better use of the administrative data already being collected and by leveraging the many policy experiments being carried out through states and districts.
- Build more and better relationships between researchers and practitioners by better utilizing existing funding and research capacity.
Below are several more specific recommendations designed to make this happen.
- Create a new IES program to train evaluators to work in school districts and state departments of education.
Like the current (and highly successful) IES program training future PhD-level academic researchers, these training programs would be based within universities. Unlike the PhD-focused program, this new training program would produce large numbers of graduates with master’s degrees and graduate certificates, developed in partnerships between universities and educational agencies that would commit to hiring graduates. These evaluators would facilitate internal research within education agencies and help build stronger bridges with external researchers.
- Provide additional competitive grant funding for states to make administrative data available to researchers and other practitioners.
As a result of legal, bureaucratic, and political barriers, it is often difficult for researchers and local education agencies to gain access to data, especially from state agencies. The federal government should provide funding to continue building more research-related capacity within state education agencies (SEAs) and require that participating SEAs make data available within reasonable timeframes and under reasonable conditions. This approach would be mutually beneficial, as universities already have the expertise to help states build and use such systems and university faculty have a keen interest in accessing the data in order to carry out their responsibilities as researchers. Michigan, North Carolina, and Texas have shown the potential of these partnerships. Others should follow suit to leverage the extensive data systems that states have already built. This, combined with the new cadre of master’s-trained evaluators, would also help change the culture of SEAs to be less focused on compliance and more focused on results.
- Change data privacy laws to facilitate fast, high-quality research.
Public concern about data privacy has grown dramatically in recent years, in part because of identity theft and regular public reports about computer hacking. As a result, state and local policies about data privacy have made it increasingly difficult to use existing education data even when the data cannot be used to identify individuals. Researchers share the concern about privacy and rarely have access to student names or any other information that would allow them to know the students’ identities. There are widely accepted ways to make data available for research and evaluation without compromising privacy, and these need to be more actively pursued. (See recommendations in the Goldhaber and Guidera memo on data privacy.)
- Shift more IES resources to research-practice partnerships.
If we followed all of the above recommendations, then it would be much easier and cheaper to also create research-practice partnerships. IES already actively supports such partnerships, but this work is just getting started and too few resources have been devoted to them. If there were more well-trained evaluators working in districts and state agencies, with better access to state administrative data systems, then it would be much easier to construct productive partnerships. This would greatly increase the potential of research-practice partnerships, warranting a significant increase in funding. Others have made the same argument. John Easton, a recent Director of IES, has done more than anyone to carry the idea forward, and the current IES team continues to effectively pursue this approach. What I am attempting to offer with this broader collection of recommendations is a way of understanding and advancing the idea going forward.
- Change the role of the Regional Education Laboratories (RELs).
The RELs are being asked to do things for which they are ill-equipped and underfunded. Federal rules (outside of IES control) make it difficult for the RELs to collect data. This, combined with the relatively short span of their five-year contracts, makes it difficult for the RELs to carry out rigorous research. Rather than carry out research they are not well positioned for, the RELs should focus their efforts on two things. First, they should reinforce their existing work to ensure that the growing quantity of studies resulting from the existing research enterprise and research-practice partnerships are easily accessible and useful for practitioners. Again, education decisions are made through relationships, and the RELs are in a position to develop and make use of such relationships. Second, the RELs should help states and districts build and use their data systems, providing localized technical assistance for leaders trying to meet the demands of ESSA, facilitating and match-making research-practice partnerships, and aiding in the deployment of the new cadre of evaluators (see recommendations 1 through 4 above). In short, the production of research should be a small part of the REL role, and they should focus more on their strengths as localized bridge-builders and facilitators of evidence-based practice. Here, too, IES has taken some positive steps and the agency should continue in that direction.
- Give higher priority to rigorous quasi-experiments.
It is common to hear researchers, and increasingly the media and policymaking community, talk about RCTs as the “gold standard.” This is misleading. Quasi-experiments, which leverage the regular policy experimentation going on in states and school districts, are, in most cases, faster, cheaper, and more generalizable to real world situations. In contrast, experiments require recruiting sites, convincing educators to carry out the intervention as designed, monitoring and enforcing treatment conditions, negotiating program and evaluation designs, and waiting for outcomes to arise. Quasi-experiments also often have better external validity because they are implemented in more realistic ways and on a larger scale that allows for rich, statistically powerful analyses of effects on different groups of students. The better quasi-experiments (especially difference-in-differences and regression discontinuity) also often share with RCTs a high degree of internal validity. Newly developed state data systems, with students’ program participation linked to their outcomes over time, make this type of work more cost-effective than ever.
To its credit, IES already recognizes the advantages and disadvantages of the various research strategies. It funds large numbers of quasi-experiments and is undertaking new programs for small, quick-turnaround studies that still maintain high standards for rigor. Also, RCTs still have an important place in some situations, especially with new program ideas that states and districts have not already attempted and where the data required go far beyond what is available in even the richest administrative data systems. That said, the agency’s requests for proposals still preference RCTs over quasi-experiments. In short, IES should help facilitate a wider public recognition of the advantages and disadvantages of different research methods, and the need to avoid letting internal validity unnecessarily drive out other important considerations like cost, speed, and generalizability.
- Increase the focus of all education research on the role of context and implementation.
While the recent focus on rigorous research has done a useful service in focusing attention on rigorous methods, we still neglect how the circumstances in which the research takes place and the specific students being served affect the results. Much greater attention needs to be paid to describing policy and practice in both control and treatment groups, providing evidence about implementation and fidelity, and analyzing effects on relevant student subgroups.
This recommendation also reinforces a larger point about research methods. Not only do we need to better balance quasi-experiments and RCTs but also to encourage qualitative methods to measure and understand the roles of implementation and context. Qualitative methods provide a richer picture of implementation and of how and why educators and students respond to programs the way they do.
As with many of these other recommendations, IES should be commended for having already taken some steps to address the concerns. This is another case where those efforts should be redoubled.
- Encourage high-quality research and development (R&D) proposals.
The above recommendations focus on evaluating the regular policy experimentation that is the strength of our nation’s decentralized education system. In addition to funding evaluations of these ideas, a key role for the federal government is funding the development of new educational programs. This is currently carried out through two main mechanisms: IES development proposals are intended as a progression in which ideas are developed, then tested for efficacy at a small scale and under ideal conditions, and finally tested under realistic conditions and at scale; and the earlier Investing in Innovation (I3) program, to be followed by the Education and Innovation Research (EIR) portion of ESSA, follows a similar logic.
There are two potential pitfalls with both of these programs. The first is that a small number of individuals or organizations will learn to “master the process” of winning the grants, limiting the potential for real innovation. Second, development projects may result in small improvements on the existing programs some organizations already run—ideas that they probably would have pursued on their own as part of the natural process of organizational development. Such outcomes run counter to real innovation and represent poor uses of federal funds. Third, and related, relatively few programs are tested at scale under real-world conditions, so it is difficult to know whether they are practical.[2]
The intent here is not to take the “D” out of education R&D but rather to ensure its success. IES and ED should give priority to proposals from individuals and organizations with a track record of success (from small-scale, rigorous evaluations to scale-up) that have not previously received such funds. These criteria, together, will ensure that new and effective ideas can emerge.
Over time, IES and ED need clear plans to test whether these federal efforts actually yield some innovative and scalable ideas. The whole idea behind development is to try bold ideas, knowing that many, if not most, will ultimately fail. This is just how R&D works. But there are steps we can take to create more success stories that eventually influence educational practice.
- Double the level of funding for IES.
By almost any measure, and despite being one of the federal government’s fundamental roles in education, the federal government is dramatically underspending in education research. Education research is an investment in long-term progress, and one the federal government is uniquely situated to make. While these proposals are designed to redirect existing funds and save money, these steps will not be enough by themselves to build the quantity and quality of research we need. The return on research funding is likely to be very high, which is why R&D has such widespread and bipartisan support.
One possible response to these recommendations is that they may go too far in shifting federal support from basic research to applied research. In other words, in trying to do research to solve immediate problems facing practitioners, we might end up doing too little to address broader questions that contribute to more general human knowledge. However, it is important to recognize two facts: First, R&D of the sort I am recommending is considered a natural bridge between basic and applied thinking. That is, we develop ideas based on basic theory and research and then apply that to the task at hand, partly as a test of our basic understanding. Second, the vast majority of federally funded education research is already applied—it is built around specific programs and policies. The real shift implied by these proposals is not from basic to applied but to more, and more useful, research.
Similarly, the goal here is not to criticize the use of RCTs or focus on establishing causal effects but to recognize the varied objectives of research and how different kinds of questions require different research methods. RCTs are especially important for studying basic scientific understanding, testing new and innovative programs, and identifying causal effects where quasi-experimental methods are unavailable or insufficient. The methods should match the questions they are attempting to answer and consider the many goals we have for research.
IES is already playing a central role in efforts to improve the nation’s education system over the long haul. While other federal agencies can no doubt also claim their importance and the benefits of more funding, it is hard to imagine a stronger case than this one. The nation and its children would clearly benefit from an increase in educational research spending, especially given how responsive and successful IES has been to date.
CONCLUSION
Education research is not an issue we will hear much about in public discourse and it is barely a blip in the federal budget, but it is important nevertheless. R&D represents a fundamental role of the federal government and one that is currently misguided and underinvested. To meet the spirit of ESSA and create a truly evidence-informed, locally driven schooling system, we need a dramatically different research-practice enterprise.
Support for this project was generously provided by the Spencer Foundation. Brookings recognizes that the value it provides is in its absolute commitment to quality, independence, and impact. Activities supported by its donors and by outside contributors reflect this commitment. The author did not report receipt of financial support from any firm or person for this memorandum or from any firm or person with a financial or political interest in this article that creates a conflict of interest.
Suggested citation:
Harris, D. N. (2016). Improving the federal role in education research. In M. Hansen & J. Valant (Eds.), Memos to the President on the Future of U.S. Education Policy. Washington, DC: Brookings Institution.
[1] The budget for the U.S. Department of Education’s Institute of Education Sciences was $600 million in 2013. Additional education research takes place within the National Science Foundation, which spent $866 million in its Education and Human Resource division in 2015, and the National Institute for Child Health and Human Development, which spent $112 million on Intellectual and Developmental Disabilities Research in 2015. General government support to universities also indirectly funds research on education, especially in colleges of education. However, these amounts also appear small. In 2013-14, total funding of public colleges was $324 billion, of which 13 percent went to research. State and federal governments funded roughly half of this total. Since governments fund roughly half of this amount, this suggests total funding for research on all subjects is about $21 billion, but only a tiny fraction of this likely goes to education research. If we assume that research funding by topic is proportional to the number of college majors, then this would suggest that 5 percent of $21 billion, or $1 billion, goes toward education research. Even this is likely an overestimate, since courses in colleges of education are generally staffed as professional schools with full-time non-research faculty. I therefore estimate total research funding, across all sources, of $2 billion annually, or 1.6 percent of the $125 billion total.
[2] Data provided by IES indicate that, in the agency’s main field-initiated grants program, 23 percent of funds go to development projects, 48 percent go toward testing existing programs and interventions, and 3 percent go toward testing effectiveness at scale (the remainder goes to measurement and exploratory analysis). The 3 percent figure significantly understates the actual level, because it excludes the large Investing in Innovation (I3) program, which falls outside of IES purview and because many of the IES-funded quasi-experimental studies are tests at scale that fall under different IES goals. Nevertheless, it does seem important for more of the IES-funded projects to eventually become evaluations of effectiveness at scale.
Authors
Commentary
Memo: Improving the federal role in education research
December 23, 2016