Increasing math skills in 5 countries: Key factors in scaling a low-tech education intervention from Botswana

The world is still coming to terms with the impact of the COVID-19 pandemic. With over a billion children experiencing some form of extended school closure from 2020 to 2022, children’s learning remains severely affected. At the same time, COVID-19 forced rapid innovation and testing new ways of doing things, including in education. Youth Impact, a nongovernmental organization based in Botswana, has been at the forefront of this response.

Since 2019, Youth Impact and partners from the Center for Universal Education have documented the process of implementing, adapting, and scaling an approach known as Teaching at the Right Level (TaRL) in primary schools across Botswana through a “Real-time Scaling Lab.” The TaRL approach delivers targeted, learner-centered instruction on basic literacy and numeracy and has been tested across multiple randomized controlled trials (RCTs). In Botswana, when school closures were announced at the onset of the pandemic, Youth Impact quickly responded to the new conditions by developing and scaling an adapted version of TaRL delivered through simple technologies with its ConnectEd program.

On average, there was a 65 percent increase in the percent of students who learned division—effectively mastering all basic numeracy concepts delivered in the course of programming.

Within days, Youth Impact and TaRL facilitators collected family phone numbers and made calls to students to enroll them in the program and assess their learning level. Then Youth Impact delivered two models over a 12-week period (Figure 1). In the first model, students received weekly math problems via SMS, targeted to their individual learning levels. In the second model, students received one-on-one, 20-minute weekly follow-up phone calls in addition to the weekly SMS.

Figure 1.

Source: Youth Impact, 2023.

Results from a study of the program showed that students who received both the phone calls and SMS messages improved their math skills during this period equal to roughly a full year of high-quality instruction gained per $100 spent. After three months, there was a 31 percent reduction in students who could not perform any numerical operations. Moreover, evaluations of ConnectEd in settings in which students had returned to school showed the program continued to be effective at improving learning even after schools re-opened.

These results prompted a coalition of NGOs, governments, multilaterals, and researchers in five other countries (India, Kenya, Nepal, Philippines, and Uganda) to test and adapt ConnectEd in their context. These trials took place between 2020-22 and seem to offer some of the most rapid and comprehensive multi-country evidence generated on an education intervention.

Results released this week from across five diverse countries are encouraging: On average, there was a 65 percent increase in the percent of students who learned division—effectively mastering all basic numeracy concepts delivered in the course of programming.

Table 1. Learning outcomes (standard deviation) across 5 countries by text only and text with phone call

Source: Youth Impact, 2023.

So, what has been behind the scale and spread of ConnectEd across a range of contexts? It appears there are many factors at play, including:

• Designing with scale and sustainability in mind from the start: The simplicity and cost-effectiveness of the approach contributed to the ease of ConnectEd’s spread. On average, the programming studied in each of the five countries cost $12 per child and equated to nearly four years of high-quality education per $100 spent, one of the most cost-effective education interviews in a recent review.
• Maintaining core aspects of ConnectEd while adapting to local contexts: Identifying core elements of ConnectEd that drive learning improvements, while maintaining flexibility to adapt the approach to local contexts, have played a key role. For example, in Nepal and the Philippines, students receiving ConnectEd were randomly assigned to one of two groups: tutored by government teachers or tutored by individuals hired by an NGO. Impacts by type of tutor were remarkably similar—the share of students who had mastered the operation differed by at most 3 percentage points across NGO and government delivery. This demonstrates that phone call tutorials can be delivered by a range of tutor profiles and supports its scalability.
• Using timely and relevant data to spur action and inform ongoing adaptations: Despite the imperative to react quickly, Youth Impact prioritized collecting data in order to understand what effect new adaptations of TaRL were having, the comparative benefits of each model, and the cost effectiveness. For example, evidence quickly revealed that despite being cheaper and more scalable, SMS messages alone were not enough to significantly improve student learning. Rather, targeted instruction delivered through interactive phone calls are the primary drivers of learning. Moreover, the speed at which evidence was generated across each of these five RCTs demonstrates that rigorous data can be generated and used to inform timely adaptation of programming.
• Seizing windows of opportunity, including moments of crisis: The disruption caused by COVID-19 provided an openness to experiment with new approaches. At the same time, ConnectEd aligns with countries’ policy priorities to focus on foundational literacy and numeracy in the early years.

TaRL’s scaling journey is still ongoing, with RCTs providing one form of insights into what is working for a specific population at a particular moment in time. While there is still much to learn, the growing evidence behind ConnectEd offers an example of how an innovation developed for one context can be adapted to and expanded in new locations through an iterative, data-driven, and collaborative process. Principles such as focusing on what already has been proven to work and applying it to areas of greatest need are ones learned during the TaRL scaling journey—chronicled through the Real-time Scaling Lab—and can help to inform other efforts to improve children’s learning at large scale.