students_math_001
Brown Center Chalkboard

The NAEP proficiency myth

Tom Loveless

On May 16, I got into a Twitter argument with Campbell Brown of The 74, an education website.  She released a video on Slate giving advice to the next president.  The video begins: “Without question, to me, the issue is education. Two out of three eighth graders in this country cannot read or do math at grade level.”  I study student achievement and was curious.  I know of no valid evidence to make the claim that two out of three eighth graders are below grade level in reading and math.  No evidence was cited in the video.  I asked Brown for the evidentiary basis of the assertion.  She cited the National Assessment of Educational Progress (NAEP).

NAEP does not report the percentage of students performing at grade level.  NAEP reports the percentage of students reaching a “proficient” level of performance.  Here’s the problem. That’s not grade level. 

In this post, I hope to convince readers of two things:

1.  Proficient on NAEP does not mean grade level performance.  It’s significantly above that.
2.  Using NAEP’s proficient level as a basis for education policy is a bad idea.

Before going any further, let’s look at some history.

NAEP history 

NAEP was launched nearly five decades ago.  The first NAEP test was given in science in 1969, followed by a reading test in 1971 and math in 1973.  For the first time, Americans were able to track the academic progress of the nation’s students.  That set of assessments, which periodically tests students 9, 13, and 17 years old and was last given in 2012, is now known as the Long Term Trend (LTT) NAEP. 

It was joined by another set of NAEP tests in the 1990s.  The Main NAEP assesses students by grade level (fourth, eighth, and twelfth) and, unlike the LTT, produces not only national but also state scores.  The two tests, LTT and main, continue on parallel tracks today, and they are often confounded by casual NAEP observers.  The main NAEP, which was last administered in 2015, is the test relevant to this post and will be the only one discussed hereafter.  The NAEP governing board was concerned that the conventional metric for reporting results (scale scores) was meaningless to the public, so achievement standards (also known as performance standards) were introduced.  The percentage of students scoring at advanced, proficient, basic, and below basic levels are reported each time the main NAEP is given.

Does NAEP proficient mean grade level? 

The National Center for Education Statistics (NCES) states emphatically, “Proficient is not synonymous with grade level performance.” The National Assessment Governing Board has a brochure with information on NAEP, including a section devoted to myths and facts.  There, you will find this:

Myth: The NAEP Proficient level is like being on grade level.

 

Fact: Proficient on NAEP means competency over challenging subject matter.  This is not the same thing as being “on grade level,” which refers to performance on local curriculum and standards. NAEP is a general assessment of knowledge and skills in a particular subject.

Equating NAEP proficiency with grade level is bogus.  Indeed, the validity of the achievement levels themselves is questionable.  They immediately came under fire in reviews by the U.S. Government Accountability Office, the National Academy of Sciences, and the National Academy of Education.[1]  The National Academy of Sciences report was particularly scathing, labeling NAEP’s achievement levels as “fundamentally flawed.”

Despite warnings of NAEP authorities and critical reviews from scholars, some commentators, typically from advocacy groups, continue to confound NAEP proficient with grade level.  Organizations that support school reform, such as Achieve Inc. and Students First, prominently misuse the term on their websites.  Achieve presses states to adopt cut points aligned with NAEP proficient as part of new Common Core-based accountability systems.  Achieve argues that this will inform parents whether children “can do grade level work.” No, it will not.  That claim is misleading.

How unrealistic is NAEP proficient? 

Shortly after NCLB was signed into law, Robert Linn, one of the most prominent psychometricians of the past several decades, called ”the target of 100% proficient or above according to the NAEP standards more like wishful thinking than a realistic possibility.”  History is on the side of that argument.  When the first main NAEP in mathematics was given in 1990, only 13 % of eighth graders scored proficient and 2 % scored advanced.  Imagine using “proficient” as synonymous with grade level—85 % scored below grade level! 

The 1990 national average in eighth grade scale scores was 263 (see Table 1).  In 2015, the average was 282, a gain of 19 scale score points.

Table 1.  Main NAEP Eighth Grade Math Score, by achievement levels, 1990-2015

Year

Scale Score Average

Below Basic (%)

Basic

Proficient

Advanced

Proficient and Above

2015

282

29

38

25

8

33

2009

283

27

39

26

8

34

2003

278

32

39

23

5

28

1996

270

39

38

20

4

24

1990

263

48

37

13

2

15

That’s an impressive gain.  Analysts who study NAEP often use 10 points on the NAEP scale as a back of the envelope estimate of one year’s worth of learning.  Eighth graders have gained almost two years.  The percentage of students scoring below basic has dropped from 48%  in 1990 to 29% in 2015.  The percentage of students scoring proficient or above has more than doubled, from 15% to 33%.  That’s not bad news; it’s good news.

But the cut point for NAEP proficient is 299.  By that standard, two-thirds of eighth graders are still falling short.  Even students in private schools, despite hailing from more socioeconomically advantaged homes and in some cases being selectively admitted by schools, fail miserably at attaining NAEP proficiency.  More than half (53 percent) are below proficient. 

Today’s eighth graders have made it about half-way to NAEP proficient in 25 years, but they still need to gain almost two more years of math learning (17 points) to reach that level.  And, don’t forget, that’s just the national average, so even when that lofty goal is achieved, half of the nation’s students will still fall short of proficient.  Advocates of the NAEP proficient standard want it to be for all students.  That is ridiculous.  Another way to think about it: proficient for today’s eighth graders reflects approximately what the average twelfth grader knew in mathematics in 1990.   Someday the average eighth grader may be able to do that level of mathematics.  But it won’t be soon, and it won’t be every student.

In the 2007 Brown Center Report on American Education, I questioned whether NAEP proficient is a reasonable achievement standard.[2]  That year, a study by Gary Phillips of American Institutes for Research was published that projected the 2007 TIMSS scores on the NAEP scale.  Phillips posed the question: based on TIMSS, how many students in other countries would score proficient or better on NAEP?  The study’s methodology only produces approximations, but they are eye-popping.

Here are just a few countries:

Table 2.  Projected Percent NAEP Proficient, Eighth Grade Math

Singapore

73

Hong Kong SAR

66

Korea, Rep. of

65

Chinese Taipei

61

Japan

57

Belgium (Flemish)

40

United States

26

Israel

24

England

22

Italy

17

Norway

9 

Singapore was the top scoring nation on TIMSS that year, but even there, more than a quarter of students fail to reach NAEP proficient.  Japan is not usually considered a slouch on international math assessments, but 43% of its eighth graders fall short.  The U.S. looks weak, with only 26% of students proficient.  But England, Israel, and Italy are even weaker.  Norway, a wealthy nation with per capita GDP almost twice that of the U.S., can only get 9 out of 100 eighth graders to NAEP proficient.

Finland isn’t shown in the table because it didn’t participate in the 2007 TIMSS.  But it did in 2011, with Finland and the U.S. scoring about the same in eighth grade math.  Had Finland’s eighth graders taken NAEP in 2011, it’s a good bet that the proportion scoring below NAEP proficient would have been similar to that in the U.S.  And yet articles such as “Why Finland Has the Best Schools,” appear regularly in the U.S. press.[3]

The Brown Center Chalkboard launched in January 2013 as a weekly series of new analyses of policy, research, and practice relevant to U.S. education.

In July 2015, the Chalkboard was re-launched as a Brookings blog in order to offer more frequent, timely, and diverse content. Contributors to both the original paper series and current blog are committed to bringing evidence to bear on the debates around education policy in America.

Read papers in the original Brown Center Chalkboard series »

Get daily updates from Brookings