Challenges and opportunities in measuring school district superintendent impact

Public school superintendents in the United States shape policy, allocate resources, and both set and work toward a shared vision for learning across the district. Surely, superintendents must influence student outcomes—but can we measure that influence?

As researchers and founders of The Superintendent Lab and the Texas Education Leadership Lab, questions like this are at the heart of our work. When we know more about how leaders impact students and schools, we can improve our policies and leadership preparation programs. Yet, ironically, while superintendents tend to be one of the most visible leaders in their communities, their effects on student outcomes can be the hardest to see—and study. 

Researchers have sought to quantify superintendent effects on student achievement, but this work faces both methodological and conceptual challenges. Quantitative studies typically use crude measures of superintendent tenure or turnover and student achievement (districtwide averages on state standardized test scores). Also, these studies do not always attend to the complex chains of influence through which superintendents might realistically affect student outcomes. This creates a risk that studies might unfairly credit or blame superintendents for test score changes driven by other factors—or fail to detect superintendent effects that operate through longer-term organizational change.

Here, we critically examine three frequently cited studies on superintendent impact: Waters and Marzano (2006), Chingos and colleagues (2014), and Redding and Carlo (2025). We then propose a path forward for researchers and policymakers seeking meaningful ways to understand superintendent influence. 

The challenge: Superintendents’ distance from classrooms

Superintendents rarely provide direct instruction to students. They operate at a distance, influencing student learning through their positional authority to affect district culture, labor relations, community and family engagement, policy development and implementation, resource allocation, and hiring and supporting administrators. This organizational distance creates three fundamental problems for researchers interested in measuring superintendent impact.

First, methodologically, even sophisticated statistical techniques struggle to isolate superintendent effects from other factors that influence student learning. In particular, there is a directionality challenge: Did test scores drop because the superintendent left, or did poor academic performance lead to the superintendent’s departure? Or maybe some exogenous factor—e.g., budget cuts or a natural disaster—caused both.

Second, how quickly should we expect superintendent impacts on student outcomes to appear? A new superintendent who hires a cohort of strong principals in one year likely won’t see those principals’ effects on teacher quality—and, subsequently, student outcomes—until several years later. This timeline challenge is compounded by the reality of superintendent tenure: According to the National Longitudinal Superintendent Dataset, just 51% of superintendents who were new to their position in fall 2020 remained as superintendent in that district in fall 2024. In other words, half of superintendents leave within four years—perhaps before their decisions have time to affect student outcomes. Even further complicating matters, some superintendent actions may boost test scores quickly while harming students over the long term (or vice versa). For example, Houston ISD superintendent Mike Miles steered some high school students into less rigorous courses where they would not take certain state exams; in turn, districtwide average test scores rose in the short-term—likely without any real academic improvement. Miles also mandated a scripted curriculum narrowly aligned to state assessments. This could have contributed to an initial rise in test scores, but evidence suggests it also drove away effective teachers.

Third, the causal pathways from superintendents to student outcomes are complex and indirect. This complexity means that researchers often lack explicit theories about the mechanisms through which superintendents impact student learning, resulting in an overreliance on vague presumptions rather than testable hypotheses.

These three challenges are interconnected. Without identifying the mechanisms through which district leaders affect learning, researchers cannot address the directionality problem or determine appropriate timelines for measuring effects. Moreover, measuring these indirect, time-lagged mechanisms would likely require collecting new data over extended periods—tracking changes in policies, personnel stability, organizational systems and more—rather than relying solely on existing administrative datasets. As a result, researchers have resorted to crude measures.

Despite these challenges, scholars have attempted to quantify superintendent impact using a range of methods and data. Here is what they have found—and what they might have missed.

Different studies, different answers

Waters and Marzano’s (2006) meta-analysis continues to be one of the most frequently cited papers for the claim that superintendents affect student achievement. However, a close examination of this paper reveals a questionable evidence base. This meta-analysis incorporated 27 studies, just three of which were peer-reviewed. Among the peer-reviewed studies—published in 1975, 1976, and 1987—two drew on one year of data from small samples (n=104 in Colorado; n=24 in Maryland) and one drew on six years of data from 66 California districts. The remaining studies were primarily doctoral dissertations (n=21), raising questions about methodological rigor. Moreover, Waters and Marzano (2006) provided few methodological details—e.g., how studies were weighted in the meta-analysis, how effect sizes were standardized across different outcome measures and research designs, or how heterogeneity across variables in the studies was assessed. Without methodological transparency or robust peer-reviewed evidence, claims from this meta-analysis rest on weak grounds. 

Chingos and colleagues (2014) offered a more rigorous analysis of superintendent impact, conducting two separate analyses. First, they employed descriptive statistics to show an “essentially non-existent” relationship between student achievement and both superintendent tenure and turnover.  Narrowing the sample to just North Carolina, they employed a hierarchical linear model and found that superintendents accounted for just 0.3% of the variance in test scores. The authors concluded, “our results make clear that superintendents have very little influence on student achievement in the districts in which they serve.” However, examining solely elementary level achievement may miss broader district impacts. Further, the short tenures of superintendents—most superintendents were in the first three years on the job—limit the ability to observe long-term impacts. Additionally, as the authors aptly note, “our methods do not support strong causal conclusion because they depend on statistical controls that are only as good as the data available to us.”

Redding and Carlo (2025) provide a recent, methodologically sophisticated analysis, using a staggered difference-in-difference model (DiD) to examine the impact of superintendent turnover on student test scores in Florida and Texas. They found a small decline in student test scores (0.015 standard deviations)—primarily driven by declines in reading scores—following superintendent turnover, particularly in urban and high-poverty districts. However, their analysis faced a critical problem: mixed evidence for the “parallel trends assumption” that underpins the internal validity of DiD designs. This assumption requires evidence that, in the absence of superintendent turnover, temporal trends in student outcomes would have been the same in treatment and control districts. The authors acknowledged mixed evidence on this point, noting significant pre-treatment differences for some groups. They note that “it is unclear the extent to which this finding should be interpreted as the causal effect of superintendent turnover.” Given these limitations, their findings are best understood as descriptive associations rather than definitive causal effects.

Finally, both Chingos and colleagues (2014) and Redding and Carlo (2025) disregarded crucial contextual differences in the nature of superintendent transitions. In particular, how a superintendent departs—an amicable resignation or retirement, a contentious firing, or a voluntary departure in the wake of school board politics—may moderate the impacts of superintendent transitions on students.

Taken together, these studies underscore the persistent difficulty in attributing student outcomes to superintendents. The studies also share a fundamental weakness: They attempt to directly connect superintendent tenure or transition to student test scores without adequately mapping the mechanisms that are at play between these two variables. This is a particular issue given the known challenges in measuring the effects that principals have on students. If we are unable to reliably measure principal effects on student test scores, attempting the same for superintendents—who are even further removed from instruction—may exceed our methodological capabilities.

A path forward: Assessing how superintendents shape conditions for learning

Measuring superintendent impact is complex and must be approached with caution. Rather than just asking how superintendent tenure or turnover impact changes in average student performance on a state standardized assessment, we should ask how superintendents shape conditions in which high-quality leadership, teaching, and learning occur. This reframing matters because research findings have consequences, and policymakers often search for evidence that confirms their preferred policies. Oversimplified research can facilitate oversimplified—and potentially harmful—policy.

Researchers wanting to study superintendent impact likely need to commit to more than rigorous data collection and analysis. This work will require time-intensive, rigorous collection of new data and the application of qualitative and mixed methods research designs. Researchers would do well to collect data and design studies that explore factors within the superintendent’s direct sphere of influence that, ultimately, impact student learning. These may include:

• Resource allocation: How, if at all, do resource allocations shift across schools, programs, and student populations when a district experiences a superintendent transition?
• Recruitment and retention of high-quality leaders: What, if any, relationship is there between superintendent stability and administrator stability and growth? How do superintendents build principals’ capacity to build strong teacher teams, and how do they support central office administrators’ capacity to support building leaders?
• Policies and systems: When a superintendent transition occurs, how does a district’s curriculum, professional development, and approach to school improvement change? How does this impact public support for the school and district climate?
• Community and family engagement: In what ways does a superintendent transition impact long-term partnerships with families, school boards, community organizations, and local agencies that support student success?

These questions acknowledge the reality of how superintendents work: through extended chains of influence that may vary by district size and context and operate on long timelines. Moreover, superintendents’ impact on these chains of influence can be fundamentally constrained by school board politics and decisions, state policies, and labor agreements.

Whether research can reliably capture the indirect, delayed effects of superintendents remains an open question. However, we are optimistic that education leadership and policy researchers can rise to this challenge, embracing the complexity required to understand how superintendents shape student learning.