Part I of the 2014 Brown Center Report on American Education
The results of the 2012 Programme for International Student Assessment (PISA) were released in December 2013. The test is administered to 15-year-olds every three years by a division of the Organisation for Economic Co-operation and Development (OECD) in Paris. Scores were reported in math, reading, and science for more than sixty nations and subnational jurisdictions. The top scoring participant, as in 2009, was Shanghai-China in all three subjects. A controversy erupted concerning Shanghai’s participation in PISA. A series of Brown Center Chalkboard essays took part in the debate.[i]
This section of the Brown Center Report will discuss the lessons that can be learned from the controversy. International tests are an increasingly important source of information on the performance of school systems. The response of PISA officials to questions raised about Shanghai highlights serious flaws in the governance of PISA that should be addressed.
Let’s start with a brief summary of the controversy.
Hukou and Shanghai’s Missing 15-Year-Olds
The Chinese hukou system is unique in the world. Started by Mao Zedong in 1958 as a tool for controlling internal migration from rural to urban areas, hukou is a household registration system that restricts rural migrants’ access to urban social services, including education. Age 15 is a pivotal year in the life of Chinese adolescents. Compulsory education ends at the end of ninth grade and students must take the zhongkao, the senior high school entrance exam, to determine their high school of attendance. Thanks to recent reforms, migrants without a Shanghai hukou can now enroll children in public primary and middle schools, but admission to academic high schools is severely restricted. In addition, students can only sit for the gaokao, the national college entrance exam, in the province of hukou registration. As a consequence, tens of thousands of families send their children back to rural villages as the children approach high school age.[ii]
The other option is to leave children behind with relatives when parents relocate to cities in search of work. Approximately 60 million children in China are “left-behinds.”[iii] An emerging literature in China documents the psychological damage done to both children and parents as families are forced apart.
Hukou is hereditary. Second generation migrants, children who are born in cities to migrant parents, are not granted an urban hukou. In 2012, a 15-year-old Shanghai girl, Zhan Haite, organized an internet-based campaign to protest being forced to attend high school in Jiangxi province.[iv] She was locked out of Shanghai’s high schools. Although she had attended both primary and middle school in Shanghai, and although she was born in Guangdong, another large city, her family’s ancestral home is rural Jiangxi. Her hukou is from Jiangxi province.
Hukou acts as a giant sifting machine, barring or driving out migrant children from urban schools. Coincidentally, PISA assesses the academic achievement of 15-year-olds. At the same time migrant families in Shanghai are feeling the brunt of the hukou system, a random sample is drawn from children attending schools in Shanghai for the PISA test. There is nothing wrong with PISA’s sampling technique, but even a pristine sampling strategy cannot compensate for a population that has already been culled of migrant students.
The impact of hukou on the 15-year-old population in Shanghai is easily seen in PISA data. Table 1-1 shows all of the participants in the 2012 PISA. On average, 15-year-olds constitute 1.2872% of the participating nations’ total population. When this average is used to predict the number of 15-year-olds in each jurisdiction—and then the countries are ranked by the deviation of the reported number of 15-year-olds to the predicted number—two outliers stand out at the bottom of the table. The United Arab Emirates’ outlier status can be explained because less than 15% of the UAE’s population are citizens—the rest are from foreign nations.[v] But notice who sits below the UAE and at the bottom of the chart—Shanghai-China. Using the 1.2872% international average as a guide, one would expect about 300,000 15-year-olds in Shanghai, a province of 23 million people. Instead, only about one-third of that amount, 108,056, is reported by PISA.
As mentioned above, migrants either leave their children behind in rural villages or bring them to Shanghai knowing that they will have to be sent away sometime before the children attend high school. The left behind population specifically related to Shanghai is difficult to pinpoint, but noted hukou expert Kam Wing Chan offers 200,000 to 300,000 as a reasonable figure.[vi] The steadily declining population of migrant children in Shanghai as they grow older can be seen in Figure 1-1, published by Professor Chan in January 2014.[vii]
PISA’s Response
Much of the Shanghai controversy involves empirical questions. Either hukou exists or it does not, either it has an effect on the migrant population in Shanghai or it does not. Surprisingly, the initial response of Andreas Schleicher, head of PISA, was to deny that hukou has any contemporary relevance in China, stating that “like many things in China, that has long changed.” Those suggesting otherwise, Schleicher charged, were embracing old stereotypes.[viii] That’s a difficult stand to maintain given the enormous amount of press coverage and academic research recently published on hukou. A 2013 report from economists at OECD, the home of PISA, condemned the hukou system for barring migrants from high school. The report states,
To attend high school it is necessary to take an entrance examination and this must be taken in the locality of registration rather than the locality of residence. In Shanghai, migrant children can only attend vocational high schools. The Shanghai Education Committee justifies local high schools’ refusal to admit the children of migrant workers on the grounds that “if we open the door to them, it would be difficult to shut in the future; local education resources should not be freely allocated to immigrant children.” As a result, few migrant children attend general high schools and those who do return to their registration locality find it hard to adapt and often fail to complete the course.[ix]
PISA’s flat out denial of hukou did not last. It then shifted into an admission that, sure, hukou still exists, but its effects are not significant. Dr. Zhang Minxuan, President of Shanghai Normal University and the Shanghai coordinator for PISA, argued that because almost 30,000 migrants were part of the population sampled by PISA (which is in line with Professor Chan’s number in Figure 1-1), no discrimination was occurring.[x] Marc Tucker and Andreas Schleicher published a response suggesting that questions about Shanghai were based on American chauvinism and envy of Shanghai’s success.[xi] All three cited China’s one child policy as cause for the shortage of 15-year-olds, although the World Bank reports that the proportion of China’s population from birth to 14-years-of-age (18%) is not extraordinary—exceeding countries such as Canada (16%), Germany (13%), Greece (15%), Japan (13%), and Korea (15%).[xii] As is clear from Figure 1-1, the one child policy is not the problem. Hukou is the problem.
Mr. Schleicher and Mr. Tucker asked, who are you to throw stones at glass houses? Schleicher pointed out that the U.S. does not include Puerto Rico in its national PISA sample.[xiii] Marc Tucker drew a parallel between the Chinese hukou system and American treatment of racial minorities and socioeconomically disadvantaged students. As he put it,
It is hard for me to distinguish the Hukou system from rural districts in the American south with white majorities who send their children to private academies and then put their fellow white citizens on the school board with instructions to impoverish the public schools serving African-American students in order to keep their taxes down, or northern communities which segregate poor and minority students into districts with no property wealth and high tax rates, while wealthy people are allowed to congregate in communities with low tax rates, gorgeous buildings and highly paid teachers.[xiv]
Schleicher, Tucker, and Zhang are honorable people. How could they defend the seemingly indefensible? Why would they be interested in downplaying the discriminatory effects of the hukou system, a policy condemned by groups such as Human Rights Watch and Amnesty International, a policy relic from China’s authoritarian past that adversely affects hundreds of millions of migrant workers who have moved from the countryside to cities, a policy that separates parents from their children, and a policy that the Chinese government itself has promised to reform?
The answer is because PISA has held up Shanghai as a model of equity since 2009. In hundreds of pages discussing equity, PISA publications have never even mentioned hukou. It is embarrassing that an entire chapter is devoted to Shanghai in the OECD’s Strong Performers and Successful Reformers in Education without a discussion of hukou. The following is said about the treatment of migrants: “In a way, Shanghai has established the notion that migrant children are ‘our children’ and works constructively to include them in its educational development.”[xv] Contrast such magnanimous sentiments with the quotation above that defends shutting out migrants.
Recommendations for the Governance of PISA
The integrity of PISA is at risk. The following three reforms would help restore PISA’s legitimacy. The PISA Governing Board (PGB) should immediately take action to implement them.
1. Ensure the Independence of the PISA Governing Board (PGB) Policy relevance has become PISA’s overarching objective, which has led to excessive and unfounded policy recommendations. It has also created an inherent conflict of interest. The OECD is a quasi-governmental body; thus, PISA is a creature of government. The PGB is made up of representatives from the governments that take part in PISA. They are not disinterested in the results.
Shanghai is a prime example of the conflict. Here is Marc Tucker describing Dr. Zhang, who, as pointed out above, coordinates the administration of PISA in Shanghai: “From 2004 through 2011, Zhang was the Vice-Director General of the Shanghai Municipal Education Committee, and, in that capacity, in charge of planning many of the education reforms for which Shanghai has since become famous.”[xvi]
Reforms in Shanghai became famous because of PISA. Shanghai’s PISA test is run by the same official who advanced the reforms. That sure sounds like a conflict of interest. Moreover, the Shanghai Municipal Education Committee is responsible for enforcing the hukou system’s education restrictions. Dr. Zhang served in a leadership capacity on that body. No wonder PISA documents are silent on the negative effects of hukou. The larger lesson concerning governance and policy recommendations applies to every PISA participant. Governments demand policy guidance from an assessment that looks at how well the policies that they themselves have enacted are functioning. It is difficult to be an impartial referee while also playing in the game.
Checks and balances need to be built into the PISA governance structure. It is unreasonable to expect national governments, who pay for international assessments, not to play a role in their governance. But that authority should be shared with educators, statisticians, policy analysts, and other independent experts from education’s disciplinary fields. The consumers of test data extend beyond government. In the U.S., for example, the National Assessment of Educational Progress (NAEP) is governed by a board that includes citizens from many walks of life and whose independence from the federal government is structurally guarded. In 1999, when Vice President Al Gore, who happened to be a prospective candidate for President at the time, appeared at a press conference to release NAEP data, the inappropriateness of the event stirred howls of outrage.[xvii]
The Trends in International Mathematics and Science Study (TIMSS) tests are governed by the General Assembly of the International Association for the Evaluation of Educational Achievement (IEA), which includes non-governmental representatives. The author of this report served as the U.S. representative to the IEA General Assembly from 2004-2012. Before him, the U.S. representatives to the IEA were from the research community (e.g., Richard M. Wolf) or worked in organizations representing state and local authorities (e.g., Gordon M. Ambach of the Council of Chief State School Officers). The PGB needs a similarly diverse membership.
2. Separate the Policy Recommenders from the Data Collectors In the U.S., the National Assessment of Educational Progress administers a national test and reports results. Analysis is primarily conducted by independent consumers of the data. Officials from NAEP or the National Center for Education Statistics (NCES) do not speculate as to why particular states score at the top of NAEP league tables or recommend that states adopt particular policies to improve their performance. It is strange that the U.S. participates in an international test that violates the constraints it imposes on its own national assessment.
The same subunit of OECD plans and administers PISA, analyzes the data, and makes policy recommendations. PISA data releases are accompanied by thematic volumes. The 2012 data, for example, were joined by volumes on equity, student engagement, and the characteristics of successful schools. The title of the 2012 volume on school characteristics reveals its ambitions: “What Makes Schools Successful?”[xviii]
These volumes are jam-packed with policy recommendations. Almost all of the recommendations are based on analyses of cross-sectional data. Skilled policy analysts are cautious in making policy recommendations based on cross-sectional data because they provide weak evidence for policy guidance.
A good example of potential pitfalls can be seen in PISA’s interpretation of the data on pre-primary education. The PISA score difference between students who, at age 15, report that they had attended pre-primary school and those who did not is large and statistically significant, 53 points before controlling for SES and 31 points after SES controls are employed. The standard deviation of PISA being 100, the gaps equate to about one-half and one-third of a standard deviation. PISA recommends more and earlier pre-primary education, part of the PISA education agenda promoted since the first PISA assessment in 2000.[xix]
Asking 15-year-olds whether they attended pre-primary school is not strong evidence on which to recommend more extensive pre-primary education. Controlling for SES does not control for all of the unmeasured variables that may be driving higher math scores at age 15. Parents who care more about their toddler’s academic success in the future, for example, may be more inclined to enroll their children in pre-primary schools—and may be more likely to emphasize mathematics when their children are teenagers. Moreover, the PISA data collectors now have taken a political position that they may feel compelled to defend. PISA officials should not advocate education policies because the data that they collect reflect on the wisdom of those policies.
3. Full Transparency on the Expansion of PISA in China PISA officials have announced plans to expand PISA testing to other provinces in China besides Shanghai in 2015. The expansion affords PISA an opportunity to dispel the secrecy surrounding the conduct of the assessment in China. Since Shanghai first participated in 2009, Andreas Schleicher has made several references in the press to PISA data collected in other Chinese provinces.[xx] But the data have never been released.
Who conducted the tests? Mr. Schleicher originally made statements such as, “We have actually done PISA in 12 of the provinces in China,” as he told the Financial Times on December 7, 2010.[xxi] After being challenged to release the data, Mr. Schleicher changed the story in an interview on Daybreak Asia to “the Chinese authorities themselves have been experimenting with PISA in other provinces, but, you know, those data are not representative so we don’t use them in our comparisons.”[xxii] Who in China took the PISA in 2009 and 2012 remains a mystery. Whether the Chinese government or OECD decided which test scores could be released also remains a mystery. Referring to 2009 PISA data from rural Chinese provinces, a 2012 BBC report stated that “The Chinese government has so far not allowed the OECD to publish the actual data.”[xxiii]
All of this needs to be cleared up before the 2015 PISA. The PGB should appoint an independent panel to investigate the conduct of PISA in China, collect relevant evidence on agreements OECD made with the Chinese government as to its participation, and release all data collected in 2009 and 2012 to the public domain. Otherwise, future PISA activities in China operate under a cloud of suspicion and doubt.
Part I Notes:
[i] Tom Loveless, “PISA’s China Problem,” The Brown Center Chalkboard, October 9, 2013, https://www.brookings.edu/blogs/brown-center-chalkboard/posts/2013/10/09-pisa-china-problem-loveless; Tom Loveless, “Attention OECD-PISA: Your Silence on China is Wrong,” The Brown Center Chalkboard, December 11, 2013, https://www.brookings.edu/blogs/brown-center-chalkboard/posts/2013/12/11-shanghai-pisa-scores-wrong-loveless; Tom Loveless, “PISA’s China Problem Continues: A Response to Schleicher, Zhang, and Tucker,” The Brown Center Chalkboard, January 8, 2014, https://www.brookings.edu/blogs/brown-center-chalkboard/posts/2014/01/08-shanghai-pisa-loveless.
[ii] Kam Wing Chan, Ming Pao Daily News, January 3, 2014, http://faculty.washington.edu/kwchan/ShanghaiPISA.jpg.
[iii] Li Tao, et al. They are also parents: A Study on Migrant Workers with Left-behind Children in China (Beijing: Center for Child Rights and Corporate Social Responsibility, August 2013).
[iv] Dexter Roberts, “Chinese Education: The Truth Behind the Boasts,” Bloomberg Businessweek, April 4, 2013, http://www.businessweek.com/articles/2013-04-04/chinese-education-the-truth-behind-the-boasts.
[v] Statistic comes from the most recent population estimate from the UAE National Bureau of Statistics, reported in 2010 and available at http://www.uaestatistics.gov.ae/EnglishHome/ReportDetailsEnglish/tabid/121/Default.aspx?ItemId=1914&PTID=104&MenuId=1.
[vi] Kam Wing Chan, Personal Interview, December 2013.
[vii] Kam Wing Chan, Ming Pao Daily News, January 3, 2014, http://faculty.washington.edu/kwchan/ShanghaiPISA.jpg.
[viii] Andreas Schleicher, “Are the Chinese cheating in PISA or are we cheating ourselves?” OECD Education Today, December 10, 2013, http://oecdeducationtoday.blogspot.com/2013/12/are-chinese-cheating-in-pisa-or-are-we.html.
[ix] OECD Economic Surveys: China 2013 (OECD Publishing, March 2013): pp. 91-92.
[x] Anthony Jackson, “Shanghai Responds to School Ranking Cheating Allegations,” Global Learning Blog, Education Week, December 21, 2013, http://blogs.edweek.org/edweek/global_learning/2013/12/
shanghai_responds_to_school_ranking_cheating_allegations.html.
[xi] Marc Tucker and Andreas Schleicher, “Response to Brookings Institution Attack on PISA,” Top Performers Blog, Education Week, December 26, 2013, http://blogs.edweek.org/edweek/top_performers/2013/12/
response_to_the_brookings_institution_attack_on_pisa.html.
[xii] World Bank, “Population ages 0-14 (% of total),” World DataBank: World Development Indicators database, http://data.worldbank.org/indicator/SP.POP.0014.TO.ZS.
[xiii] Andreas Schleicher, “Are the Chinese cheating in PISA or are we cheating ourselves?” OECD Education Today, December 10, 2013, http://oecdeducationtoday.blogspot.com/2013/12/are-chinese-cheating-in-pisa-or-are-we.html.
[xiv] Marc Tucker, “Tom Loveless on Hukou in China,” Top Performers Blog, Education Week, January 24, 2014, http://blogs.edweek.org/edweek/top_performers/2014/01/tom_loveless_on_hukou_in_china.html.
[xv] “Lessons from PISA for the United States,” Strong Performers and Successful Reformers in Education (OECD Publishing, 2011): pp. 96.
[xvi] Marc Tucker (Ed.), Chinese Lessons: Shanghai’s Rise to the Top of the PISA League Tables (Washington, DC: National Center on Education and the Economy, 2014), pp. 2.
[xvii] John Stevens, Getting the Word Out About the National Assessment: The National Assessment Governing Board and Changes in NAEP Reporting (National Assessment Governing Board. March 2009): pp. 7.
[xviii] PISA 2012 Results: What makes School Successful? Resources, Policies, and Practices (Volume IV), (OECD Publishing, 2013).
[xix] PISA 2012 Results: What makes School Successful? Resources, Policies, and Practices (Volume IV), (OECD Publishing, 2013).
[xx] Chris Cook, “Shanghai tops global state school rankings,” Financial Times, December 7, 2010, http://www.ft.com/intl/cms/s/0/20770bf6-01e7-11e0-b66c-00144feabdc0.html; C. M. Rubin, “The Global Search for Education: Can You Pass the Global Standardized Test?” The Huffington Post, May 17, 2011, http://www.huffingtonpost.com/c-m-rubin/global-standardized-testing_b_862165.html; Sean Coughlan, “China: The world’s cleverest country?” BBC News, May 12, 2012, http://www.bbc.co.uk/news/business-17585201.
[xxi] Chris Cook, “Shanghai tops global state school rankings,” Financial Times, December 7, 2010, http://www.ft.com/intl/cms/s/0/20770bf6-01e7-11e0-b66c-00144feabdc0.html.
[xxii] Jim Stevenson (Narr.), Daybreak Asia, December 10, 2013, https://www.brookings.edu/blogs/brown-center-chalkboard/posts/2014/01/08-shanghai-pisa-loveless.
[xxiii] Sean Coughlan, “China: The world’s cleverest country?” BBC News, May 12, 2012, http://www.bbc.co.uk/news/business-17585201.
| « Introduction | Part II: Homework in America » |
