Following up on previous Brookings research measuring the value colleges add to student outcomes irrespective of student characteristics, this study analyzes the Obama administration’s new College Scorecard database to produce value-added rankings for 3,173 colleges (1,507 two-year colleges and 1,666 four-year colleges), based on the earnings of alumni.
Value-added measures attempt to isolate the contribution of the college to student outcomes, as distinct from what one might predict based on student characteristics or the level of degree offered. It is not a measure of return on investment, but rather a way to compare colleges on a more equal footing, by adjusting for the relative advantages or disadvantages faced by diverse students pursuing different levels of study across different local economies.
Using the Scorecard data and other sources, this analysis finds that:
1. Graduates of some colleges enjoy much more economic success than their characteristics at time of admission would suggest. Colleges with high value-added in terms of alumni earnings are often focused on training for high-paying careers in technical subjects. These colleges include the Albany College of Pharmacy and Health Science, the Massachusetts College of Pharmacy and Health Science (MCPHS), The University of Texas Health Science Center at San Antonio, all of which have a strong medical orientation, and Maine Maritime Academy, SUNY Maritime College, and the California Maritime Academy, which focus on nautical engineering and related subjects. Two-year colleges with high value-added scores include the North Central Institute, ATS Institute of Technology, Lawrence Memorial Hospital School of Nursing, and CUNY Kingsborough Community College.
2. Four college quality factors are strongly associated with higher earnings for alumni:
Curriculum value: The amount earned by people in the workforce who hold degrees in a field of study offered by the college, averaged across all the degrees the college awards;
STEM orientation: The share of graduates prepared to work in STEM occupations;
Completion rates: The percentage of students finishing their award within at least 1.5 times the normal time (three years for a two-year college, six years for a four-year college);
Faculty salaries: The average monthly compensation of all teaching staff
3. Value-added measures are fairly reliable over time. Scores for the cohort entering in 1997–1998 strongly predict value-added for the cohort entering in 2005–2006, using earnings data measured six-years after admission. Nonetheless, some colleges have made large improvements on value-added over that period, including Stanford, Yale, Georgetown, and Emory, as well as Glenville State College and Alaska Pacific University.
No ranking system is perfect, and there are many limitation and caveats with these data. Yet, the value-added measures here help fill an information void and can be used to inform strategies to improve college quality and help identify schools that are contributing the most to student economic advancement. There are many colleges with modest or low admission standards that consistently prepare students for high income careers. The next step in college quality research is to attempt to measure learning gains.
Explore the new data
For two-year schools, top value-added performers also tend to have a technical orientation. North Central Institute in Tennessee emerges as the top value-added performer on 10-year median earnings for federal aid recipients entering in 2001 and 2002. Actual earnings are 56 percent higher than predicted for attendees, who often study aeronautics and take courses in applied physics. A number of colleges focusing on nursing score very highly, including ATS Institute of Technology, Lawrence Memorial School of Nursing, St. Vincent’s College, Laboure College, Southeast Missouri Hospital College of Nursing and Health Science, and the Carolina College of Health Sciences. Meanwhile, students at Bramson ORT and Foothill College typically study business or information technology.
Value-added measures are fairly reliable over time.
One of the most novel and compelling aspects of the Scorecard database is that it reports earnings data for multiple cohorts of entering students. The earliest data available cover the 1997 to 1998 cohort, while the latest apply to the 2005–2006 entering classes. Value-added measures are calculated for each, and the change is reported below.
The changes in value-added can be decomposed into changes in the observable measures of quality. Unfortunately, in terms of shedding light on college quality, most of the changes are driven by changes in unobservable factors (the x-factor). This suggests there is still much work to be down to measure observable and predictable aspects of college quality.
Still, even with so much unobserved, these value-added scores are broadly stable over time. There is a high correlation between value-added for the 1997–1998 cohort and value-added for the 2005–2006 cohort for four-year colleges (0.82). For two-year colleges, value-added is notably more volatile. The correlation is just 0.58 for the two periods. Medium-term value-added (i.e. 10-year earnings for the 2001–2002 cohort) appears somewhat more robust in that correlations with the 1997–1998 cohort and the 2005–2006 cohort are higher for both two- and four-year colleges.
Despite general stability in the measure, some colleges showed large increases in short-term value-added from 1997–1998 to 2005–2006
While broadly stable over time, value-added scores for individual schools sometimes changes dramatically. Below, the colleges with the largest increases are highlighted, with a view toward summarizing some of the broad factors behind the increase.
Among four-year colleges, value-added increased the most for the Palmer College of Chiropractic in Davenport, Iowa. It increased the earnings of alumni, even as predicted earnings fell. The average family income of incoming students receiving federal aid was $50,000 in 1997–1998, corresponding to 0.6 standard deviations above the mean four-year college. But by 2005–2006, the mean earnings of incomes students was just $13,000, 1.7 standard deviations below the mean. Remarkably, earnings increased from 0.8 standard deviations below the mean to 0.5 standard deviations above the mean.
Some well-known schools also showed large increases in value-added between these periods, including Stanford, Yale, Georgetown, Duke, Emory, and Harvard. For each of these colleges, predicted earnings increased slightly over the period, even as the federal aid recipients at these colleges were from further down the income ladder in 2005-2006 than in the previous period. This would imply that these colleges are increasingly drawing from less economically advantaged families, but this conclusion does not necessarily follow. At Harvard, for example, the percentage of students receiving federal aid fell from 44 percent in 1999 to 24 percent in 2005, making the Scorecard data less relevant in the later period than in the former. These data do not tell us about the majority of students at Harvard who do not receive federal aid.
For some of the most improved colleges, value-added simply became less negative. This was the case for Central Yeshiva Tomchei Tmimim Lubavitz, the California Institute of the Arts, Washington Bible College-Capital Bible Seminary, and Manhattan Christian College. As implied by the names of these colleges, maximizing the earnings of graduates is probably not a principal goal for the administrators of these schools or their students. Rather, the focus is on preparation for careers as religious ministers and artists. Still, graduates from the later period have made some economic progress relative to their peers.
As for two-year colleges, some of those with the largest increase in value-added went from extremely poor performing to merely below average. This describes Stone Child College and Aaniiih Nakoda College, both located in rural Montana, Cossatot Community College of the University of Arkansas, Sisseton Wahpeton College in South Dakota, and Windward Community College in Honolulu. With the exception of Windward, these schools serve very low-income students in low-income areas. At Stone Child College, the average family income of students entering in 2005-2006 was just $15,000. The increase in Stone Child College’s curriculum value—placing it near the top of community colleges—explains some of its increase in value-added. However, modest their most results may be, these schools have made substantial progress for their students and communities.
Other schools with large improvements went from slightly above or below average to excellent, including the New Mexico Military Institute, North Central Texas College, and Lamar Community College in Texas. The same could be said of Whatcom Community and Bellingham Technical College, both in Bellingham, Wash.
Background
Earlier this year “Beyond College Rankings: A Value-Added Approach to Assessing Two- and Four-Year Schools” aimed to present and validate new measures of the market value of the skills taught at specific colleges and improve how colleges are ranked The research, of which I was the lead author, did so by comparing diverse colleges on a more level-playing field and by calculating the economic value they add to alumni earnings and related outcomes. The value-added calculation compares expected alumni outcomes to actual outcomes, rewarding colleges that do the most for the least advantaged students.
As noted in the study, there were many data limitations and reasons for caution. The earnings data came from an organization called Payscale, which relies on voluntary data entries from users of its salary negotiation website. While Payscale offered the most useful and comprehensive source of earnings available at the time, those data contained little coverage of 2-year colleges, unknown sample sizes, limited age-cohort comparability, a non-random sample of alumni, and no information on non-graduates.
As I’ve described elsewhere, the Obama administration’s new College Scorecard database represents a substantial improvement in data quality and transparency in higher education. Its core earnings measures are highly correlated with Payscale data for those colleges covered by both sources, but it contains a number of advantages: 11.3 million alumni compared to 1.4 million in Payscale; coverage for roughly 5,800 institutions compared to just 1,300 for Payscale, with deeper coverage per college; precisely defined age-cohort metrics to maximize comparability; and detailed data documentation with clearly articulated and well-defended methods. Moreover, the Scorecard contains novel measures of the family incomes and family education levels of entering students and more sophisticated measures of degree completion and loan repayment than those previously available.
The Scorecard’s weaknesses, however, are non-trivial. It only reports earnings data for students starting as undergraduates who received federal aid—either loans or grants, which applies to roughly half of all students, and federal aid recipients tend to have lower family incomes than their peers. Moreover, some colleges with multiple campuses report have earnings data only at the system level, not at their specific campuses. This affects roughly 8 percent of students in the Scorecard database.
A larger issue is that alumni salary, when taken alone, is a biased and imprecise measure of the quality of a college, or even its economic value. It is biased because colleges with high earning alumni also tend to have students who are better prepared academically—as measured by test scores—and come from higher-income families. These students would have been well positioned to earn at least moderately high salaries, even if they attended a less prestigious college. In short, many factors contribute to alumni salary outcomes that have nothing to do with the quality of the college, but much to do with the characteristics of the students. This is why a value-added approach is needed.
Of course, salary is not the same as quality. The most relevant qualities of a college will vary by individual. It may include something about the depth and importance of the material taught, insights gained, skills mastered, sense of purpose established, and friendships forged, none of which can be directly measured by earnings. Researchers should strive to collect comparable institutional data on these factors.
Nonetheless, many critics of the College Scorecard and similar attempts to analyze alumni outcomes go too far in rejecting the importance of salary information, as my colleague Beth Akers has argued. Clearly, earnings are substantially relevant to happiness and health, even as there are many other determinants of those goods. At a social level, the public and its government representatives have a very strong interest in other people earning higher salaries, because those salaries lead to higher tax revenues, elevated investment in research, culture, and the arts, new goods and services, and increased spending that sustains economic growth and prosperity.
This report takes advantage of the new database to provide greater insight into the strengths and weaknesses of U.S. colleges in preparing students for remunerative careers. Using methods developed in “Beyond College Rankings,” this report analyzes the determinants of earnings for thousands of colleges for students entering in 1997–1998, 2001–2002, and 2005–2006 and compares these to value-added measures using Payscale.
After a brief description of the method, the report shows—at the aggregate level—which observable quality factors predict value-added and how they compare using Payscale and College Scorecard data. Then, college level value added results using medium-term 10-year earnings are reported for four- and two-year schools. (In the appendix, these rankings are compared to Payscale in an effort to better understand the strengths and weaknesses of the two sources and identify colleges that do well on both). Next, the analysis turns to changes in value-added over shorter-term horizons, comparing the stability in rankings overall and showing which colleges have improved the most. The final section concludes with a brief discussion of the limitations to current sources of college data. The Appendix describes the technical analysis and details about how to compare Payscale earnings with the Scorecard.
An updated method
The companion Methodological Appendix explains the details of the method and analysis used here.
The theory underlying this analysis is that the future earnings of alumni are affected by student characteristics (such as their academic preparation, age, racial or ethnic background, and family income), the type of college (a community college or research university, for example), the location of the college (as in a big city with many jobs compared to a small town), and the qualities of the college (Figure 1). To estimate the college’s contribution to student earnings, earnings for an individual college are predicted based on these attributes and compared to actual outcomes. The value-added of the college can be thought of as the difference between expected and actual outcomes, or the sum of unmeasured and measured qualities of the college.
Key outcomes
This study focused on three earnings measures from the College Scorecard ( Table 1). One complication of any outcome measure is when to measure it. Lifetime earnings are best approximated by earnings later in life, but that provides less relevant information to current cohorts of entering students, who face a different labor market and may be entering a school that has changed significantly. Research on tax records for millions of Americans finds that annual earnings tend to reflect lifetime earnings very well by age 30. As it happens, the Council of Economic Advisors finds very high correlations at the college level between six-year and 10-year earnings measures, suggesting that both provide relevant information about students’ lifetime earnings potential. Hence, this report focuses on median earnings six and 10 years after entry to college.
Table 1. Earnings outcomes used in this study
| Measure of earnings | Year of entry into college |
| Median annual earnings 10 years after start of college, measured in 2011 and 2012 | Students entering in 2001 and 2002 |
| Median annual earnings 6 years after start of college, measured in 2011 and 2012 | Students entering in 2005 and 2006 |
| Median annual earnings 6 years after start of college, measured in 2003 and 2004 | Students entering in 1997 and 1998 |
Notes: Earnings data are for employed, non-enrolled, undergraduate students who received federal aid (loan or grant) and who reported earnings to the IRS, likely excluding students living abroad. Change in earnings calculations adjust earnings to 2014 dollars using the consumer price index. Students who do not graduate, transfer, or go on to get a subsequent degrees at any institution are included if they are working not enrolled during the years of data collection.
For comparison to previous work and as a check on the robustness of the data, this report also estimates value-added using the Payscale 2015-2016 College Salary Report for mid-career salaries (see Appendix and companion interactive and database). This measure assigns median earnings to a school from its pool of graduates (not dropouts) who have not gone on to earn a higher degree and who graduated at least 10 years ago from the time of measurement. For four-year colleges only, Payscale also provides the same measure for all undergraduate alumni (including those pursuing higher degrees).
A college’s qualities that contribute to its value-added
As in earlier work, this analysis distinguishes between observed and unobserved qualities of a college.
Currently, there is no database available that would allow one to measure many aspects of a college that are likely related to its quality, such as teacher quality, administrative competence, and the utility of student programs. These unmeasured attributes of a college we refer to as its X-factor.
Still, there are many things about a college that can be observed, such as the extent to which its curriculum aligns with high-paying fields of study (its curriculum value and STEM orientation), how well its keep students on track (its graduation rate), and how well it compensates its teaching faculty. See the Appendix of “Beyond College Rankings” for details on how curriculum value and STEM orientation are measured).
The sum of observable and unobservable qualities of a college—weighted by their predictive power in explaining earnings—constitute the college’s value-added.
Variables that contribute to a school’s predicted outcomes
Calculating value added requires comparing predicted salaries to actual salaries. The prediction is driven by characteristics of the college’s students and the college itself that are unrelated to how well it prepares students for economic success. These characteristics are listed below and include test scores, family incomes, and the regional cost of living near the college.
Findings
A college’s curriculum value, mean faculty salary, and completion rates strongly predict the future earnings of those who attend.
Colleges have no effect on what students learn before they enter, but they do control the content of what students learn while enrolled. Colleges also control faculty salaries, subject to market pressure and, for some public colleges, state legislation, affecting their ability to recruit and retain top teachers. Finally, colleges also affect degree completion, at least to some extent, through summer programs, financial aid packages, tutoring services, and other less tangible aspects of the academic environment.
As our previous analysis showed, these factors are strongly predictive of higher earnings after graduation (Figure 2). Among the quality measures, curriculum value is the single most important factor. A standard deviation in curriculum value is worth an estimated 11 percent in higher annual earnings using Scorecard data and 7 percent using Payscale data. In general, the quality variables predict higher earnings using either source.
The characteristics of students, of course, also affect earnings. A standard deviation in the average family income of entering students adds 22 percent to predicted medium-term earnings. Conventional rankings, based on selectivity and other measures, and non-value-added rankings of colleges based on earnings, fail to capture this important relationship, and thus overrate colleges with high-income students and underrate those with low income students.
Colleges with the highest medium-term value-added for federal aid recipients are often focused on training for high-paying careers in technical subjects.
Among four-year colleges, those with the highest value-added for federal aid recipients are disproportionately specialized in high-paying fields of study, such as medicine, business, and engineering. Thus, Albany College of Pharmacy and Health Sciences reports the highest value-added score. Actual median earnings–$116,000—are more than double predicted earnings, based on student characteristics and other factors. It is followed by the Massachusetts College of Pharmacy and Health Science (MCPHS). Both schools prioritize health-related fields like molecular biology, chemistry, pre-medical and nursing. The University of the Sciences in Philadelphia trains many students for careers as pharmacists, among other health-related occupations. Three other top scoring colleges—Maine Maritime Academy, Massachusetts Maritime Academy, and SUNY Maritime College—prioritize engineering related to shipping logistics and offshore drilling. Meanwhile, Babson and Rose-Hulman—focus on business and engineering respectively.
Among the top 20, only Harvard and Duke rate highly on conventional rankings like U.S. News & World Report’s. On the other hand, Brigham Young University, which ranks 66th in U.S. News, ranks 16th on value-added for medium-term earnings.
Conclusion
Overall, the College Scorecard represents a major advance in the effort to transparently compare colleges along dimensions of quality. It’s major limitation—the lack of any value-added metric—is addressed here and in the companion interactive. Yet, it makes strides toward allowing college administrators, policy officials, and others responsible for the quality of a college to more readily identify and resolve its weaknesses, with a view to enhancing student well-being after attendance. It also can help make the market for higher education function more efficiently via greater transparency and consumer empowerment.
However progress is severely limited by flaws in available data, as well as for more fundamental reasons. The remaining flaws in the scorecard and other sources include:
1. Incomplete coverage. In the Scorecard’s case, this means the data are limited to federal aid recipients. For Payscale, it means a non-random sample and many institutions with no data. Neither report outcomes for students by graduate institution or program of study.
2. Lack of outcomes data disaggregated by field of study. This is true for all multi-state databases, but disaggregated earnings are reported by Texas Consumer Resource for Education and Workforce Statistics for alumni who remain in the state of Texas.
3. No measures of learning. Perhaps the serious omission related to data on college quality is the absence of any information about what students learn while in college. While achieving higher earnings are often a secondary goal, acquiring a body of useful or meaningful knowledge is perhaps the chief goal of higher education. The OECD is working to develop metrics that could be used to measure learning in higher education, but comprehensive measures seem a long way off.
More fundamentally, there will always be serious imperfections to any effort to assess college quality. People are not randomly assigned to colleges and differ in important ways that will remain unobservable to any researcher.
Given these limitations, some contend that colleges simply cannot be meaningfully ranked or compared, but as we have seen in the above (and in elementary and secondary value-added analyses by leading economists) reasonably accurate and reliable measures of value-added can be produced for individual schools, especially four-year colleges. Such measures should never be mechanically and rigidly tied to fiscal or regulatory policies, but they can and should serve as relevant guideposts to how well specific institutions are doing in preparing students for economically prosperous lives.
Authors
