Course correcting learning in the provision of skills

Growing labor market segmentation in the West and the perceived political and social benefits of youth employment in other parts of the world have increased the demand for public programs to improve worker skills. Recent comprehensive reviews of skills development programs, however, show poor returns and emphasize the importance of better program design. Impact evaluation techniques are argued to be a useful learning tool to improve policy rollout. Pakistan Punjab’s ongoing Punjab Skills Development Fund (PSDF) has utilized these techniques to increase the impact of skills training. The results are encouraging. 

In response to growing fears of militancy spilling over into Punjab from the restive tribal areas bordering Afghanistan, the government of Punjab and the United Kingdom Department for International Development (DFID) set up a 50-million-pound skills development fund to improve employability of vulnerable young adults in the four poorest districts of South Punjab. The five-year program was given two objectives: (i) train 145,000 vulnerable young men and women from relatively poor income groups to acquire skills that increase their income; and, (ii) given poor performance of the established public training programs, develop private sector capability to provide training.

The fund was launched in 2011, as a not-for-profit government company. It has an independent, private sector dominated board that provides policy oversight and assesses company performance. Operations are carried out by a market hired CEO and his team of competitively selected employees.

The PSDF board added a third objective to introduce a culture of course correcting learning to meet program objectives. This entailed carrying out rigorous ongoing evaluations of all critical stages of the training process to ensure that the two PSDF objectives are met cost effectively. This was made possible due to a unique collaboration between PSDF and a local research center (CERP) that drew on leading academics from Harvard, Princeton, the Institute of Development and Economic Alternatives (IDEAS) and the Lahore University of Management Sciences. The evaluations span demand for training modules, content of the courses offered, private training firms’ training bids, the quality of course delivery, trainee attendance, and the income impact of training. Endorsing the third objective, DFID and Punjab government added a well-funded technical assistance component (1.5 million pounds) to the program. This, and the fact that leading academics have turned their attention to rigorous evaluations for improving development policy outcomes, has resulted in a robust culture of learning that informs the rollout of PSDF training modules.

The largest impact from evaluations is at the first and the last stages of the training rollout, i.e., a correct assessment of the demand for training and the impact of training on trainee income. In the poor districts, the baseline survey revealed strong household demand: 90 percent of the households responded that they will enroll at least one young adult, male or female. This was followed up by the offer of vouchers, a stipend of 1000 rupee (about $10) per month per trainee, for the four month long courses and information on the location of training centers. Since the vouchers could only be cashed at the end of the training month, the young adults would need to commit to the training sessions in order to receive payment. But, in the last stage evaluations, it was found that only 47 percent of the households took up the vouchers. Of those, only 5 percent enrolled in the program. The strong demand expressed earlier for training did develop into strong enrollment rates. 

Based on information gathered from households in a subsequent survey on the reasons for the low uptake, the research team carried out a pilot small-sample randomized controlled trial to compare three interventions: (i) standard information to households on the objectives of the training program and its design; (ii) community-based social mobilization to encourage enrollment; and (iii) in-village training. Female potential trainees were targeted because they had the lowest uptake and enrollment rates. The results (see Figure 1) are encouraging. Compared to the standard procedure (information only), social mobilization managed to raise enrollment (completion) by over 6 (14) times, while village-based trained had an even more dramatic effect – increasing enrollment (completion) by over 13 (31) times! These results have led to subsequent design refinements of the program  by introducing additional interventions such as group transport to address access constraints faced by women. [1]

Figure 1: Training course enrollment, and completion by treatment type

Ongoing assessments of the impact of training on income also suggest that interventions that link the trainee (who acquires skills to produce for the market) to the buyer are needed to realize the full welfare gains of training. While the impact evaluation is ongoing preliminary trainee   tracer surveys revealed that the provision of training alone can increase income. However, initial results suggest that this increases further with interventions that bring the trainee closer to the market by facilitating subcontracting arrangements with buyers.  The income impact of training is therefore likely to be substantially higher with further deepening of these links, once established.

PSDF completed its five years in June 2016. A total of 147,599 trainings have been delivered (the target was 145,000). Female trainings were 38 percent of the total (the target was 40 percent). This would not have been possible without the course correcting evaluations that improved the design of the program. Total private training companies engaged by the program to deliver training increased from 29 companies in 2011-12 to 214 companies in 2015-16. The operational cost of the program, capped at 7 percent, was under 5 percent.

What’s more, PSDF has helped initiate a culture of evidence-based policymaking in the government. Punjab Chief Minister Shehbaz Sharif has given PSDF board the flexibility to experiment with new approaches to service delivery so that the successful experience can be emulated by other government departments. In addition, the government of Punjab and DFID have agreed to scale up the program to span all of Punjab doubling the size of the fund to 100 million pounds for the next five years.

[1] Crossing Boundaries: Experimental Evidence on Access Constraints for Rural Women” (A. Cheema, A. Khwaja, F. Naseer, and J. Shapiro, CERP Working paper, 2017)