student_Shanghai003
Report

PISA’s China Problem Continues: A Response to Schleicher, Zhang, and Tucker

Tom Loveless

In October 2013, I posted an essay, “PISA’s China Problem,” that called on the Organization for Economic Cooperation and Development (OECD) to fully disclose its arrangement with China regarding Shanghai’s participation in the Program for International Student Assessment (PISA).  The latest PISA scores were to be released in December, offering an excellent opportunity for the OECD to dispel the mystery surrounding Shanghai’s 2009 involvement with PISA. I noted that Shanghai, the wealthiest, most educated province in China, was the only mainland province officially participating in PISA 2009 and PISA 2012.  Other data from rural areas of China had been talked about by PISA officials over the years, but never released to the public domain.  I called on PISA to release those data.

When the latest PISA scores came out in December, nothing had changed.  I followed up with a second essay.  I again urged full transparency.  I also challenged PISA’s portrayal of Shanghai as a “high equity” school system.  An extensive literature—including excellent journalism and both qualitative and quantitative scholarship–documents the cruel effects of the hukou system on migrants in Shanghai.  Hukou is an internal registration system in China that limits rural migrants’ access to urban public services, in particular, to schools.  These migrants are Chinese citizens, mind you, not immigrants from other countries.  They have simply moved from rural areas to China’s big cities, or, because the hukou is inherited, they were born in one of China’s big cities but because of their family’s rural hukou, have become second generation migrants in the eyes of the state. 

Andreas Schleicher of OECD-PISA wrote a response to my essays, as did Dr. Zhang Minxuan, President of Shanghai Normal University.  Marc Tucker, President and CEO of the National Center on Education and the Economy, joined Mr. Schleicher in publishing a third response in Education Week.  

I answer them here.  I organize the discussion by the two main questions in dispute.


Issue #1: Is Shanghai a Model of Equity?

OECD-PISA has portrayed Shanghai as a model of equity since the province began participating in PISA in 2009.  In the foreword to the 2009 volumes analyzing PISA results, Shanghai is the first PISA participant mentioned by name, singled out for attaining high scores and high equity with a diverse population of students:

The education systems that have been able to secure strong and equitable learning outcomes, and to mobilize rapid improvements, show others what is possible to achieve.  Naturally, GDP per capita influences educational success, but this only explains 6% of the differences in average student performance.  The other 94% reflect the potential for public policy to make a difference.  The stunning success of Shanghai-China, which tops every league table in this assessment by a clear margin, shows what can be achieved with moderate economic resources in a diverse social context.[1]

This statement, testifying to the power of public policy, is unintentionally ironical.  It ignores the hukou system, a public policy condemned by human rights groups such as Human Rights Watch and Amnesty International, and that China itself labors to change.  It ignores how the hukou system has a devastating impact on the lives of migrant children. It ignores how the hukou, by design, shows disdain for social diversity.  It ignores how, even today, this cruel but powerful public policy influences who attends Shanghai’s public schools by driving out migrant children. 

In their responses to my essays, Dr. Zhang, Mr. Tucker, and Mr. Schleicher spend a good deal of time defending PISA’s sampling procedures.  That’s a diversion.  I explicitly stated that “this is not a sampling issue.”  The issue is the truncated 15-year-old population that remains in Shanghai once PISA sampling commences.  In the December post, I presented a table of population data from ten countries and Shanghai showing that the other PISA participants with about the same number of 15-year-olds (Shanghai reported 108,000 in 2012) have national populations that are only one-half to one-third the size of Shanghai.  How can a country the size of Portugal, population 10.6 million, produce almost the same number of 15-year-olds as Shanghai, population 23.0 million?

Dr. Zhang calls this approach naïve and unscientific.  He raises the valid point that the ten  comparison countries are economically developed and most are in Europe. 

Table 1 rectifies that shortcoming.  It includes all PISA participants except for Lichtenstein, which was omitted because its total population is less than 50,000 (thereby rounding to zero under the table’s rounding protocol), a total of 64 jurisdictions.  It includes developed and developing nations and countries with birthrates even lower than China’s.  The first column shows each country’s national population.  The second column displays the total number of 15-year-olds in the country as reported by PISA.  The third column (“15-Year-Olds % of POP”) shows the percentage of 15-year-olds as reported by PISA expressed as a percentage of each country’s overall population.  The world average is 1.2872%. 

The multiplicative inverse of 1.2872% is approximately 80, meaning that 80 is the divisor one would use to produce a ballpark estimate of the number of 15-year-olds in a country when only that country’s total population is known.  The fourth column uses the world average to predict the number of 15-year-olds, the fifth column (Difference) reports how far off the number of actually reported 15-year-olds on PISA is from the predicted number, and the last column expresses the difference as a percentage of the predicted number.  This method of predicting countries’ 15-year-old populations is quite robust.  The correlation coefficient for the predicted value and that reported by PISA is 0.964.

Table 1 rank orders the PISA participants by the last column.  There are two outliers at the bottom of the table, Shanghai and the United Arab Emirates (UAE).  These two participants under-report fifteen-year-olds in comparison to all other PISA countries.  

 Table 1.  Analysis of 15-Year-Olds in PISA

National POP (mil)

15 Year Olds Reported by PISA

15 Year Olds % of POP

Predicted Number of 15 year olds

Difference

PISA’s 15 year olds as % of Predicted (C/E)

Albania

3.2

76,910

2.40344

41,191

35,719

186.72%

Jordan

6.1

129,492

2.12282

78,520

50,972

164.92%

Peru

29.5

584,294

1.98066

379,729

204,565

153.87%

Vietnam

86.9

1,717,996

1.97698

1,118,593

599,403

153.59%

Colombia

45.5

889,729

1.95545

585,684

304,045

151.91%

Malaysia

28.3

544,302

1.92333

364,283

180,019

149.42%

Mexico

112.3

2,114,745

1.88312

1,445,546

669,199

146.29%

Brazil

193.3

3,574,928

1.84942

2,488,193

1,086,735

143.68%

Costa Rica

4.6

81,489

1.77150

59,212

22,277

137.62%

Indonesia

237.6

4,174,217

1.75683

3,058,431

1,115,786

136.48%

Turkey

73.7

1,266,638

1.71864

948,680

317,958

133.52%

Argentina

40.5

684,879

1.69106

521,323

163,556

131.37%

Chile

17.1

274,803

1.60704

220,114

54,689

124.85%

More