To make ed tech work, set clear goals, review the evidence, and pilot before you scale

Child at a computer
Editor's note:

This blog is a summary of the report, “Realizing the promise: How can education technology improve learning for all?

Over the past few decades, we have witnessed multiple predictions that technology would  “disrupt” education (remember when the $100 laptop was going to revolutionize education in the developing world?). The current COVID-19 pandemic has been no exception. Yet, the reality of integrating technology into our school systems has been sobering, to say the least: Some innovations have demonstrated new ways for students to learn, but few have been implemented at scale, and most have had small to moderate effects on achievement.

Why is there such a disconnect between the prophesized potential and the results of “ed tech?” The eagerness with which the private sector has promoted some hardware and software, the incentives that politicians face to adopt popular (but not necessarily effective) reforms, the frequent use of technology to reproduce regular “chalk-and-talk” instruction (instead of playing to what it does best), and an unclear focus on how technology seeks to improve children’s daily experiences in school all play a role.

How can we realize the potential of ed tech to improve student learning? In a new report with Rick Hess, we argue for a simple yet surprisingly rare approach to education technology that seeks to: (1) understand the needs, infrastructure, and capacity of a school system; (2) survey the best available evidence on interventions that match those conditions; and (3) closely monitor the results of innovations before they are scaled up. A unifying theme throughout our report is that those interested in realizing the potential of education technology should think carefully about how it will improve the learning process.

The diagnosis: How can school systems assess their needs and preparedness?

A useful first step for any school system to determine whether it should invest in education technology is to diagnose its: (a) specific needs to improve student learning (e.g., raising the average level of achievement, remediating gaps among low performers, and challenging high performers to develop higher order skills); (b) infrastructure to adopt technology-enabled solutions (e.g., electricity connection, availability of space and outlets, stock of computers, and internet connectivity at school and at students’ homes); and (c) capacity to integrate technology in the instructional process (e.g., students’ and teachers’ level of familiarity and comfort with hardware and software, their beliefs about the level of usefulness of technology for learning purposes, and their current uses of such technology).

Before engaging in any new data collection exercise, school systems should take full advantage of existing administrative data that could shed light on these three main questions. Yet, for those systems without available data, we developed a set of surveys for students, teachers, and principals. The surveys encourage system leaders to: (a) identify their main student-learning challenges (e.g., ensuring that all students reach minimum standards, helping the lowest performing students catch up with their peers, or looking for ways to encourage them to acquire higher order skills); (b) take stock of their available infrastructure to deploy technology-enabled interventions (e.g., physical infrastructure, internet connectivity, and hardware at schools and students’ homes); and (c) assess the degree to which principals, teachers, and students are ready, willing, and able to integrate technology into instruction (including not only their technical competence, but also their belief about its usefulness).

The evidence: How can school systems identify promising ed-tech interventions?

An important next step in the process of assessing the potential of investing in education technology is to take a close look at the best available evidence on ed-tech interventions. To assist decisionmakers in this process, we identify four potential comparative advantages of technology to improve student learning in developing countries, including: (a) scaling up standardized instruction (e.g., through prerecorded lessons, distance education, or preloaded hardware); (b) facilitating differentiated instruction (e.g., through computer-adaptive learning or one-on-one tutoring); (c) expanding opportunities for practice (e.g., through practice exercises); and (d) increasing student engagement (e.g., through video tutorials or games and gamification).

Yet, just because technology can do something, it does not mean it should. School systems around the world differ along many dimensions, including their size, level, and distribution of students’ skills, and the capacity of their public sector bureaucracy to implement reforms at scale and of teachers to deliver high-quality instruction. To inform the decisions of policymakers in different types of school systems, we review in detail 37 impact evaluations of ed-tech interventions across 20 low- and middle-income countries. Rather than recommend any single intervention across contexts, we provide education officials with useful illustrations of where some interventions have worked best (i.e., what need was addressed) and why they worked (i.e., what was the theory of change for improving instruction, as well as the evidence to support it).

The prognosis: How can school systems adopt interventions that match their needs?

A crucial final step is to experiment with the most promising innovations (based on the match between the needs of the school system and available rigorous evidence) and to monitor implementation and results closely before scaling. While this may seem obvious, many ambitious ed-tech reforms failed because they required drastic changes in a system’s infrastructure (e.g., in internet connectivity) or because of low uptake in schools. Understanding why innovations fail is crucial not only to prevent systems from scaling up ineffective interventions, but also to iterate on their design, and address problematic issues. A successful ed-tech intervention is one that addresses an important learning need with current levels of infrastructure and capacity, has an evidence-based theory of change, and is seen as useful and thus implemented as intended by principals, teachers, and students. Iterative piloting and rapid feedback are key to identifying the intervention that meets these conditions, and can help bridge the disconnect between ed tech’s prophesized potential and actual results.