This month’s Powerball winners collected $1.6 billion, but not all lotteries turn out well for winners. In Louisiana, students who won a lottery for tuition scholarships to private schools wound up with worse academic performance than their peers who were lucky enough to lose the lottery.[i]
The affected students had won a voucher to attend, at no cost, a private school in Louisiana. Nineteen states have such voucher programs, with Louisiana’s the fifth-largest in the country. The vouchers, averaging $5,311 per student, must be accepted as full tuition at the private schools that participate in the program; schools are not allowed to ask students to “top-up” their vouchers if the school has a higher sticker price. Further, schools can’t pick and choose among the voucher winners. Instead, they have to take any student who holds a voucher.[ii]
Nationwide, 141,000 students use a voucher to attend a private school.[iii] Louisiana’s voucher program launched in New Orleans in 2008. It was expanded to include the entire state in 2012. Students from families with incomes below 250 percent of the federal poverty threshold are eligible for the voucher as long as they attend a public school the state has labeled as low-performing. Over half of Louisiana’s public schools fall into this category.
Researchers have long attempted to understand the effectiveness of private schools. It’s a difficult task, because parents choose their children’s schools, either by living in a certain school district or by applying to a private or charter school. The challenges are identical to those in evaluations of charter-school effectiveness: kids who attend private school are different from those who attend a public, neighborhood school, who in turn are different from those who attend a charter school.[iv] When comparing school performance, researchers struggle to distinguish differences in schools’ effectiveness from variation in the types of students who choose those schools.
A voucher lottery provides an unusual opportunity to measure the effectiveness of private schools. The lottery serves as a randomized trial, which is the gold standard of research methods. Random selection means that lottery winners and losers are identical, on average, when they apply for the voucher. Any differences that emerge after the lottery can therefore be attributed to the private-school attendance of the winners.
The results were startling. The researchers, a team of economists from Berkeley, Duke, and the Massachusetts Institute of Technology, found that the scores of the lottery winners dropped precipitously in their first year of attending private school, compared to the performance of the lottery losers. The effects were very large: roughly a quarter of a standard deviation in math, social studies, and science. There were no effects on reading scores. On a per-year basis, these negative effects are as large as the positive effects that a similarly-designed study found for charter schools in Boston (the authors of the Louisiana study are my collaborators in the charter research).[v]
It’s possible that the students entering private schools simply had a difficult transition year, and that’s why their scores plummeted. But similar studies in charter schools, as well as other voucher studies, have found no such negative, first-year effect. The only way to know if the effect is positive or negative in the long term is to continue to track these students (using administrative data) as they progress through school. The researchers say they hope to carry out this follow-up research, assuming that the state continues to make the necessary data available.
It’s also possible, as the researchers speculate, that the private schools participating in the voucher program are of lower quality than other private schools in Louisiana. They note that the private schools participating in the program experienced sharp enrollment declines right before they joined the program, which may indicate that paying parents found these schools unsatisfactory. Private schools outside the voucher program saw no similar declines.
At the very least, the results suggest that the participating private schools need to provide far more support for voucher students when they enter. If the voucher students continue to perform poorly, Louisiana needs to overhaul the criteria used for including schools in the voucher program—or shut down the program altogether.
While these negative results are troubling for the students, and for education policy in Louisiana, they hold a silver lining for social science research. Many worry that scientists tend to publish only positive results, burying studies that show negative or null findings. This is definitely a problem in medical research, where pharmaceutical companies disproportionately release the results only of trials that show their drugs to be effective.[vi] This has led to a push for registries of experiments, where researchers publicly post their intention to conduct a study and commit to a research methodology.[vii] A similar movement is afoot in the social sciences, where randomized trials are becoming more common.
Previous research that uses lotteries to study schools has almost uniformly yielded positive results, with the occasional zero effect. The Louisiana study bucks that trend, and that’s a sign that science is working as it should. Science and policy move forward by learning from successes as well as from failures.[viii]
[i] http://economics.mit.edu/files/11259
[ii] https://www.louisianabelieves.com/schools/louisiana-scholarship-program
[iii] http://afcgrowthfund.org/yearbook/
[iv] http://www.nytimes.com/2015/11/22/upshot/a-suburban-urban-divide-in-charter-school-success-rates.html
[v] http://www.nytimes.com/2015/11/22/upshot/a-suburban-urban-divide-in-charter-school-success-rates.html
[vi] http://www.scientificamerican.com/article/trial-sans-error-how-pharma-funded-research-cherry-picks-positive-results/
[vii] http://www.alltrials.net/find-out-more/all-trials/
[viii] https://www.brookings.edu/research/papers/2015/12/03-harnessing-value-failure-jacob
