Helping foster smart data use in a small state

Today, there are frequent calls for quality analytics and “data-driven decisionmaking.” Indeed, in a time of scant resources, policymakers often understand administrative data sets as the only means available for informing their work on tight timelines. But, the pressure to find quick answers to questions that are often very complicated can lead to oversimplifying education challenges, trends and issues, sometimes with confounding results. Without understanding the particular contexts from which the data come, what the data are capable of telling us, and where their limitations lay, we run the risk of coming to erroneous conclusions that have real consequences for practitioners, students, families, and communities.

In Vermont, tiny, declining enrollment and convoluted governance structures create additional hurdles to sound statistical analysis. With a population fewer than 700,000 people and a student body of around 85,000 during the 2015 school year, the Green Mountain State boasts 277 school districts and 14 different forms of governance.  Additionally, in 2014-2015, nearly 80 percent of schools serving up to 6^th grade enrolled fewer than 300 students, while nearly 70 percent of schools serving 9-12^th grades enrolled fewer than 500. Due to such conditions, Vermont has had difficulty assessing and reporting outcomes and matters of equity for student sub-group populations. In most schools, these sub-groups have so few students that their data must be suppressed for privacy reasons when reporting. In addition, minor changes in class composition potentially yield large but not substantive shifts in reported performance, rendering meaningful statistical analysis all but impossible (e.g. one student may be eight percent of the subgroup).

Historically, this has presented the state with an opacity bemoaned by lawmakers and the public alike, especially when questions of equity, access, and fiscal sustainability were on the table. So, in 2015, Governor Peter Shumlin made data a priority. In response, the Vermont Agency of Education (VT AOE), my employer, launched an expansive effort to improve data use and literacy both across the education system as well as among legislators, stakeholders, and the media. In tandem, we embarked on designing and implementing a state accountability model that uses statistical data sets alongside qualitative approaches to evaluation to yield methodologically robust, highly contextualized information about our education system.

Education Quality Reviews

We call these processes Education Quality Reviews (EQR). EQR take a multi-pronged, mixed-methods approach to education system evaluation. On an annual basis, administrative data sets will be mined from our State Longitudinal Data System (SLDS) to create indicators that reflect the statistical backdrop of Vermont’s schools and supervisory units across five domains of operations. We call these statistical portraits Annual Snapshot Reviews (ASR), and will use them to examine quality and equity, both within and across sites.  These numerical measures will be complimented by in-depth, on-site, qualitative and quantitative evaluations called Integrated Field Reviews (IFR). IFRs are designed to provide the rich, nuanced details that shed further light on the trends made visible through ASRs, because even the best statistics will not show the whole picture. In this way, both measures will contribute to better understanding areas where schools have made progress, where there is room for growth, and just as importantly, provide our systems with some feedback on how they can improve.  

The integrated approach of EQR is geared toward helping address the challenges of Vermont’s size conditions and complex governance structures by ensuring both appropriate use of quantitative data and that those data are properly situated within qualitative data that provides the context to fully understand them. It also accounts for a broader conception of learning and education than that captured by standardized testing alone. And, by bringing practitioners together across sites to observe, gather data, identify effective practices, and provide feedback in the field reviews, we expect the process to strengthen the capacity of our educators in making systems improvements. With our Education Quality Reviews, we are trying to shift the conversation, empower our practitioners, and support more meaningful data use in both the creation of smart policy and in the provision of equitable, excellent education for Vermont’s children.

While there is much work to do ahead, efforts like EQR, which bring thoughtful, sound methodological approaches to tough empirical problems in the applied environment, represent a shift taking place in the practitioner community across the country. As state longitudinal data systems mature and data begins to play a more central role in decisionmaking at the policy and the classroom levels, careful attention to the power and the pitfalls of these data are needed now more than ever. Taking conscientious approaches to measurement for evaluation and accountability, collaborating with stakeholders to devise appropriate methods, exploiting the value of mixed-methods approaches, and committing to improving data literacy in schools, among administrators, and in policy arenas will help make the education environment of tomorrow not just data-driven, but data-smart and data-strong.