The Supreme Court and the Perils of Advocacy Science: Examples from the Schuette Oral Argument on Affirmative Action

The Supreme Court has a history of getting itself in trouble when it too readily turns to social science and statistics to bolster its legal decisions. The oral argument in the recently submitted Schuette case offers two examples of how the Court may be led astray. Citing the work of UCLA law professor Richard Sander, who submitted an amicus brief in Schuette, Chief Justice Roberts suggested that maybe in banning affirmative action Michigan’s voters were acting in the interest of the state’s minorities and saving them from the harm of “academic mismatch.” Later Michigan’s Solicitor General John Bursch, responding to a question about the harm to diversity that the ban on affirmative action is likely to bring about, gave answers that can only mislead a conscientious Court.

The mismatch hypothesis that Roberts is willing to buy has an intuitive appeal. It makes sense that students admitted to competitive schools with academic credentials (mainly test scores and grades) considerably lower than those of their peers would feel overmatched and flounder as a result, and affirmative action students get, on average, worse grades than their peers. But this does not mean they are better off if they are directed to less selective institutions. Not only is the empirical evidence for mismatch scanty and flawed, but considerable evidence calls the hypothesis into question and indicates that if one looks at post-school achievements “reverse mismatch” seems to prevail; that is students who attend more selective institutions do better than similarly credentialed students who attend schools with peers who are more like them on admissions credentials.

“…students who attend more selective institutions do better than similarly credentialed students who attend schools with peers who are more like them on admissions credentials.”

One reason Supreme Court Justices go astray is that they may attend to only one side of an academic debate, and even when they are aware of two sides they will lack the technical skills to choose between them. Much of the support for mismatch that Justice Roberts referenced and which Justice Thomas alluded to  in his dissent in Fisher v. Texas, last year’s affirmative action case, is the work of Professor Sander and an economist and erstwhile coauthor Douglas Williams.  But their major contributions, even if otherwise flawless, could tell us almost nothing about whether affirmative action leads to harmful mismatch today because it is based on law student data from the Bar Passage Study (BPS), now more than 20 years old.  When the data were collected, 41.4% of entering African American law students had admissions index scores above 600 and 22% had scores below 500 (on a 1000 point scale). In the 2010 entering cohort, 60.5% had index scores above 600 while those with index scores below 500 had all but disappeared (5.7%). Looking just at LSAT scores, 419 matriculating African American law students had scores of 160 or above, three to four times the number of African American students with such scores when the BPS data were collected. This change is fundamental because as economists Jesse Rothstein and Albert Yoon pointed out, “moderately qualified students do not appear to experience mismatch effects even when they attend highly-selective law schools.”   A Justice reading the Sander and Williams corpus would be unlikely to appreciate the significance of the datedness of their core data, particularly since the most recent article by the latter appeared in June 2013.    

Moreover, the empirical case for mismatch is far from flawless; it has not in fact stood up to scrutiny. Although Professor Sander’s original article is still cited by himself and others in support of law school mismatch, almost as soon as it was published he admitted to serious flaws in its data and methods and proceeded to try to resurrect his conclusion using different methods, and by reliance on the work of Williams.  This effort did not succeed. At least half a dozen independent scholars looking at the same data with a variety of methods failed to find evidence of mismatch effects that, if strong, should have emerged. This includes two recent papers by Peter Arcidiacono, another erstwhile Sander co-author. Looking at the University of California system, he finds the evidence is inconsistent and where it favors mismatch effects are small. Looking at Missouri system data he finds reverse mismatch effects in that holding credentials constant African American students do better when they attend the system’s most selective institution. But the recent work should not be necessary to quiet doubts. These should have been resolved in Fisher when 11 social scientists, most of whom had no prior involvement in affirmative action debates, reviewed the key studies in the amicus brief that Sander, as a coauthor, offered the Court. This group, which included Donald Rubin and Gary King, both members of the National Academy of Sciences, along with several other of the country’s top social science methodologists, advised the Court that the empirical evidence cited to it as favoring mismatch should be given no credit whatsoever. In his Schuette amicus Sander both reoffered empirical evidence he had offered in Fisher and claimed that the social science critique in Fisher had fundamentally erred. Some Justices might entertain Sander’s claim, but they would not know that it had been earlier presented to his Fisher critics, and they saw no reason to change their conclusion.

To my knowledge since Fisher only one article that purports to offer empirical support for the mismatch hypothesis has been published in a peer reviewed journal, a piece by Douglas Williams offered to the Court as a draft in Fisher but in print  by the time Schuette was argued. Although surviving peer review is one cue a Court should rely on in assessing empirical research, it doesn’t guarantee soundness. Even competent peer reviewers can go wrong, and this is particularly likely when they are reviewing the analysis of unfamiliar data. Williams did several different analyses, but the strongest evidence for mismatch results from a comparison of the performance of African American students attending law schools in the top two and bottom two of the study’s six law school tiers. Not only does this comparison eliminate more than half of the African Americans in the data set (discarding data is seldom good practice) but Williams misleads for more fundamental reasons.

A feature of the data set, perhaps reflecting unmeasured social characteristics of Tier 2 law schools, is that African Americans in this tier do worse, relative to similarly credentialed students in other tiers in graduating and passing the bar. If rather than combining tiers one and two, as Williams does, one looks only at African Americans in top tier (Tier 1) schools, differences are in the opposite direction. No matter what their admissions credentials, African American students do as well as or, in most cases, better in graduating and passing the bar than similarly credentialed students at less competitive institutions, and overall they did extraordinarily well. Few students leave top tier schools before graduation and well above 90% of those taking the bar pass. These results, which defy the predictions of the mismatch hypothesis, are obscured when the first and second BPS tiers are combined.

Further bias with respect to the mismatch hypothesis results from combining the two bottom tiers to provide a single comparison group. Three-quarters of the African Americans in the combined group are in sixth tier schools, but Tier 6 is unlike the other tiers. It is not defined by variables like mean admissions credentials and the public/private dichotomy. Tier 6 is instead composed solely of historically black law schools which at the time the data were collected were on average more than 40% African American. Controlling for admissions credentials African American students in historically black law schools do particularly well in graduating and passing the bar. But this is not necessarily because they experience lesser mismatch; in fact more than any other tier these historically black law schools have students with admissions credentials at all levels. The obvious alternative explanation for their success is that the cultural comfort that comes from attending schools with a “critical mass” of African Americans. Unless explanations of this sort can be dismissed, the relative success of African Americans at these schools cannot be safely attributed to lesser mismatch. Williams admitted as much in a forum at the Brookings Institution a year ago. A Justice reading this study would have no idea about these confounds or about how the decision on which tiers to combine and compare makes results that could mistakenly be seen to support mismatch more likely.

Also missing from the case for mismatch is the evidence that matters most; namely evidence on how well the beneficiaries of affirmative action do after they graduate. Here the evidence appears clear and consistent. African Americans who have been able to enter the nation’s most selective schools due to boosts from affirmative action do well upon graduation, and the more selective the school the better they seem to do. This is the conclusion of Bowen and Bok’s well known study, The Shape of the River. Consistent with Bowen and Bok’s findings, when I and two coauthors looked at the careers of affirmative action minorities who had attended the University of Michigan Law School over a 27 year period, we found that 90% and more of the minorities attending Michigan graduated and passed the bar. When we followed their careers through time we found that despite substantial admissions “mismatch,” the school’s minority alumni earned as much as its white alumni, were as satisfied with their careers and gave more back to the community in the form of leadership and legal work done pro bono.   A study by Davidson and Lewis of the University of California at Davis medical school graduates reached consistent conclusions.  The most methodologically sophisticated of the career outcome studies is recent work by two economists, Stacy Dale and Alan Krueger. Looking at the earnings of students who graduated college in 1976 and 1989, they find, that with certain“[n]otable exceptions” attending more selective colleges, somewhat to their surprise, had no noticeable effect on future earnings. Unlike whites, black and Hispanic students did far better in future earnings when they had attended more selective institutions. Not only do these findings run counter to the mismatch hypothesis, but they also indicate that the earnings gains that accrue to minorities from gaining admission to selective schools do not come at the expense of any white students they displace.

It is not just social science in briefs that can mislead the Court.  Attorneys in oral argument can do so as well. Again Schuette provides an example. When the Michigan Solicitor General, John Bursch, was pressed on the degree to which the Michigan prohibition on affirmative action had depressed minority enrollments at the University of Michigan (UM), his response was evasive and misleading. He suggested that nothing could be known about how “Prop 2,” which placed a ban on affirmative action in Michigan’s constitution, had affected UM’s diversity because consistent with the 2010 Census UM changed its forms to invite  students to indicate a multiracial heritage, rendering, the Court was told, the before and after Prop 2 data incomparable. But comparing the undergraduate enrollment of “underrepresented” minorities in 2006 with enrollments in 2012, there is a drop of 15% even if everyone indicating mixed ethnicity is considered an affirmative action minority and a drop of 27% if they are not. At the graduate level, the main gateway to academic and professional careers, the effects on diversity is yet more devastating. The 2012 graduating class included the lowest percentage of Masters Degree recipients since 1989 and the lowest percentage of doctoral degree recipients since 1993. The proportion of African Americans getting law degrees in 2011 and 2012 was down by about 60% from its 1990s and 2000s average, and in 2012 the medical school graduated only 5 African American MDs, percentage wise a decline in the production of African American physicians of almost 70% as compared to the years 2004-2010. The Court heard none of this.

Similarly misleading was Bursch’s attempt to bolster the notion that affirmative action is essential for diversity by reference to the California experience. To this end he told the Court that minority enrollments at 16 of 17 University of California (UC) campuses have increased since California’s Proposition 209 outlawing affirmative action. His figures are puzzling since the UC system contains 10 campuses, one of which was established after Proposition 209, and its California State University system has 23 campuses. How Bursch chose his statistics a mystery. Yet more misleading is what was left unsaid. California minority enrollments have held up numerically to the extent they have because there has been no drop in  proportion of California high school graduates who are African American while the proportion of Hispanics have skyrocketed and on many campuses enrollments have increased.  

To give a success story, at UC Davis, the average number of enrolled freshman who are African Americans during the three most recent admissions years (2010-12) is about a third higher than it was during the three years prior to the effective date of Proposition 209 (1995-97). What these figures don’t tell us is that both before and after Proposition 209 African Americans have constituted about 3% of the Davis freshman class, far less than their proportionate representation in California’s high school freshman classes. To put this another way, between these dates UC Davis added about 1130 students to each new entering class. Of these new places only 38 went to African American students. The increase is minimal despite the enrollment advantage Davis gains from African Americans who can no longer enter the system’s most selective institutions.

At UCLA, one of the nation’s most selective public universities, the picture is quite different. During the three years prior to the affirmative action ban, UCLA freshman classes had on average 260 African American who had applied as California residents. In the most recent three years this average has fallen to 185, or a bit more than 70% of the former total even though there were about 700 additional freshmen in each year’s entering class. This drop occurred despite a movement to holistic admissions, numerous outreach efforts and the institution of California’s 4% plan. With Hispanics there has been an absolute increase in the number of residents in each freshman class, from an average of 683 in the three years before Proposition 209 went into effect to 893 in the three most recent years, but after taking the increase in the size of UCLA’s freshman class into account the percentage increase in Hispanics starting as freshman is less than .7%. Moreover, while this small increase was occurring, the average number of Hispanics applying to UCLA rose from 3703 a year to 12,734; the chance that an Hispanic applicant would be admitted to UCLA fell from 42% to 14.5%, and the proportion of Hispanic applicants who enrolled as freshmen fell from 18% to 7%. It is only by lumping African Americans with Hispanics and neglecting to note the huge increase in the proportion of California’s high school graduates who are of Hispanic heritage that Mr. Bursch was able to make it appear that banning affirmative action might have relatively minor implications for the number of underrepresented minorities in state flagship schools like the University of Michigan.

The implications of all this for Schuette is that whatever their decision Justice Roberts and his colleagues should not delude themselves into thinking that the white citizens of Michigan were, when they outvoted the state’s minorities to ban affirmative action, intentionally or by happenstance acting to prevent harm to minority students or engaging in an exercise without serious consequences for enrollment diversity at Michigan’s state-supported professional schools and its most selective universities.

Editor’s Note: Professor Lempert was one of the 11 signers of the amicus brief in Fisher referenced in this post, although he neither wrote the brief nor does he include himself among the signers he characterizes as methodological experts.  He is also is one of three lawyers who together submitted an empirically-focused amicus brief on behalf of the respondent in Schuette.