A more expansive approach to studying what works in education

For many years, the U.S. Department of Education’s Institute of Education Sciences (IES) has been tasked with helping answer a simple, important question: What works in education? That is, which programs, interventions, and services help kids learn?

In particular, the What Works Clearinghouse (WWC) defines what makes for a quality evaluation, including which statistical methods allow us to say with confidence that a program improves outcomes for students. On one hand, this evaluation focus in education is somewhat niche, representing a fraction of the entire field of education research. On the other hand, IES and the WWC define best practices for evaluating educational programs and interventions, establishing the gold standard. As a result, this niche area holds major sway.  

In its first year, this Trump administration has made large cuts to the IES budget. Not only have many staff been laid off, much of the funding that IES provided for grants is uncertain, cut off, or both. As a result, some of the top evaluation researchers in the country, working in think tanks and policy shops, lost their jobs. The field of education evaluation research is in turmoil.

Like most crises, this one should not go to waste. Previously, I worked as a fiscal analyst for the State of California. That job began in 2008-09, at the heart of the fiscal downturn. I asked a colleague how she survived a job that involved cutting education budgets day in and day out. She replied simply that moments of crisis also represent moments of reinvention.

We need to use this moment in educational research to reinvent what it is to understand educational effectiveness. As I will argue, we currently approach educational efficacy in a way that is far too narrow. We need something more than a clearinghouse to help identify what works in education; we need a similar focus on understanding how and why something works—and when it is unlikely to work at all.

Today’s empirical education research is far too narrow

I’m a psychometrician by training, focused on educational and psychological assessment as it relates to program evaluation. In many ways, these two fields—educational evaluation and assessment—are the main culprits in narrowing how we understand effectiveness. Such disciplines have created a hyper focus on two pursuits: 1) determining causality (i.e., how an education program or policy affects student outcomes); and 2) focusing on outcomes with readily available quantitative measures, like test scores. The two are linked. Analyses identifying what causes what in education, by necessity, rely on large-scale quantitative data like test scores.

However, the evaluation field so fetishizes causality—which relies on easily available quantitative data—that we often lose sight of the larger picture. We study whatever program allows for a randomized control trial or natural experiment, and we focus results on the narrow outcomes we can measure, like achievement. Too often, we let the study design drive what we evaluate rather than the other way around. Education, to many evaluation researchers, has become a construct winnowed down to almost nothing. (This issue isn’t unique to education, either.)

I first became skeptical of quasi-experimental studies while serving in that fiscal analyst role in California. Researchers would come to Sacramento to present their newly published regression discontinuity or difference-in-differences study. Many of these studies were carefully done, but many also were detached from the reality of policy implementation, such as how long it takes to actually implement the policy. These anomalies made it hard for me to trust the black box of quasi-experimental results and what came out of them.

Yet, my main concern isn’t about what most of these studies find. It’s about what they omit.

First, they omit an exploration of mechanisms, which should be the lifeblood of educational research. To identify causal effects, many studies only compare intervention results for students to outcomes of kids in the same district, school, or even classroom. To me, this approach misses the point. Educators don’t just want to know the effects of a particular intervention. They want to know why an intervention worked in specific contexts—and how it could work in their own.

Second, they omit outcomes that are too nuanced to straightforwardly quantify. This point was made to great effect by Eve Ewing in “Ghosts in the Schoolyard,” a book about school closings in Chicago. Decisions were made based on easily quantifiable measures like test scores and attendance—a primary focus of empirical research. While these measures are important, policymakers paid much less attention to considerations like the role of the school in the community, what it means to families, and how students would feel going to another school. I would add to that list other vital outcomes that could relate even more directly to whether kids thrive, such as critical thinking skills, socioemotional competencies, and psychological well-being. These outcomes cannot be easily, cheaply measured, but they matter.

Quantitative data feel impartial and fair. But that objectivity is only realized when the data show the whole picture. Focusing heavily on achievement and attendance in Chicago wasn’t necessarily wrong, but it severely narrowed the aperture of reality.

A new approach for the future

In many ways, what we need is more causal research, but much more research that broadens the lens. We need more qualitative research, especially where it works in tandem with, and corroborates, causal results. We need an emphasis on mechanisms. (Here, we might learn from the field of psychology, which gives real credence to mechanisms and has produced the methods to test hypotheses about them.) We need measures that go beyond achievement and are valid for their intended purpose. We need to try to understand truth, not only cause, whenever we can. We need research that understands contexts and people.

We can apply this thinking to IES and what it might look like going forward. For example, the best practices of the future might involve publishing experimental or quasi-experimental results in tandem with qualitative or survey-based studies on the same intervention. The quasi-experimental study would hypothesize and model the causal mechanism at play, and the qualitative study would consider whether that mechanism seems valid, as well as what contexts might undermine or enhance that impact. This new approach to understanding quality would hopefully incentivize cross-disciplinary research that gets at the why of what works. It would help explain which interventions are, and are not, portable to other contexts.

Being more attentive to context and conditions—to the “how” and “why” of what works in education—also represents a beginning (if not an end) to addressing criticisms that education research is too hard to translate to concrete actions and decisions by policymakers and practitioners. If we can better pinpoint mechanisms, we can better help educators on the frontlines tell if a strategy tried elsewhere is likely to benefit a particular group of kids, in a particular context, facing particular types of challenges.

I still believe that IES and all its endeavors—including the WWC—have served a vital role in education. We should still try to identify what works in education. But we should pair the WWC with a how and why it works clearinghouse. Let’s not let this crisis go to waste.