Series: Evidence Speaks
Report

Hard thinking on soft skills

Grover J. “Russ” Whitehurst

Executive Summary

The nation’s PK-12 education ecosystem is poised to embrace programs intended to enhance soft skills. Soft skills are generally defined by exclusion as personal qualities other than the formal knowledge transmitted by schools that affect student adjustment, i.e., the effort that students put into their work and their social skills. Such soft skills are far too important for the education reform effort associated with them to suffer the fad-like fate of far too many education reforms of the past. There are danger signs in that regard.

One problem is that advocates of soft skills reform have approached the conceptualization and measurement of soft skills in ways akin to how psychologists approach human personality, i.e., as relatively enduring, trait-like individual differences in broad patterns of behavior. Such patterns of behavior are highly heritable, meaning that schools will have difficulty influencing differences among students. They are also abstract and general, meaning that they provide little of the specificity that is needed for the design of curriculum for students in different grades or for the provision of useful feedback to teachers or students. Further, the theory and measurement of soft skills in schools is in its infancy, with many critically important questions unanswered.

Also troubling are recent research findings that charter schools that are both effective in raising student achievement and focused on character development either have no impact or a negative impact on students’ self-reported soft skills. Such findings conflict with the implicit theoretical model of soft skills reform in which the causal path to better academic achievement and life outcomes flows through students’ soft skills as enhanced by schools.  

A prudent way forward for educators given the many acknowledged unknowns in soft skills reform is to substantially enhance efforts that fall within traditional school practices and responsibilities rather than to boldly make risky bets on unproven programs and measures. Practical steps for school and district administrators include: 1) focusing on improving student behavior, not personality traits; 2) implementing schoolwide rule systems focused on respectful social interactions; 3) using measures of soft skills that are naturally occurring and useful as feedback at the classroom and individual level; 4) establishing priorities around students who are significantly off-track in their social-emotional behavior or self-management skills; 5) establishing priorities around remediation or removal of teachers whose interpersonal behavior toward students is likely to be doing harm; and 6) putting in place systematic ways to learn from and improve the reform efforts.

Background

You live on the north coast of England in the 10th century. A lookout, seeing the longboats make landfall, shouts the alarm—“the Vikings are coming.” The fiercest warriors in the western world are about to descend on you. Your panic and fear are unimaginable.

Cut to the 21st century. Someone says “the Scandinavians are coming.” You feel mild unease because you don’t like high taxes and jellied fish. Rape, pillage, and plunder are not among your concerns.

It doesn’t take a difference of a 1,000 years in Nordic culture to make the point that human interpersonal behavior is malleable and consequential. I travel a lot. If I don’t consciously remind myself where I am, the chances of faux pas rise substantially. I might, for example, make eye contact and say good morning to a stranger on an elevator in Manhattan, or fail to do so in Raleigh. Over the years and many school visits I’ve been struck more than once with conspicuous differences between how students and adults behave in buildings that look pretty much the same and serve similar students—orderly vs. chaotic is the essence of the contrast.

These time and place examples support the intuition that interpersonal behavior of groups of people is a product of cultural institutions. This belief, with a dash of evidence thrown in, is the bedrock of the advocacy movement for the elevation of soft skills in the education curriculum.

Surely soft skills are important and schools have an important role in shaping them. But the reality is that research on soft skills is soft. It isn’t even clear what we’re talking about (stay tuned), much less what works in schools that are trying to improve student competences in this domain, or who should be held accountable for what and how.

It is also important to understand that individual differences in soft skills are as or more important than the central tendencies of cultural and group differences: some Vikings were more aggressive than others; among people trying to be friendly on an elevator, some pull it off effortlessly and others strain; and the orderly school contains some students who are tumultuous. The rank order of individuals on dimensions such as aggressiveness and sociability is relatively stable even as the forms of expression vary with time, place, culture, and circumstance. This has implications for the priorities and focus of effort by schools trying to impact soft skills.

The nation’s PK-12 education ecosystem seems poised to embrace programs intended to enhance soft skills. In part, this is due to a new requirement in the recently reauthorized federal Elementary and Secondary Education Act (ESSA) that state accountability systems include at least one nonacademic measure. Nine school districts in California that serve roughly 1 million students have already organized as the CORE Districts and instituted an accountability system in which 40 percent of the total weight for school success is assigned to measures of social-emotional outcomes,[i] including a component based on direct self-report assessments of students on growth mindset, self-efficacy, self-management, and social awareness.[ii] Also notable are eight large urban districts scattered across the country that are part of the Collaborative for Academic, Social, and Emotional learning (CASEL).[iii] They are in the process of carrying though on a commitment to adopt social and emotional learning standards, put in place social and emotional learning programs for students and professional development programs for teachers, and carry out social and emotional learning assessments.

Soft skills are far too important for students for the education reform effort associated with them to suffer the fad-like fate of far too many past education reforms, including those that were based on a valid core assumption. The goals of the present report, which will extend over subsequent reports, are: first, to raise important questions in the context of the expansion of efforts by schools to enhance the soft skills of their students and measure outcomes; second, to suggest what prudent school officials and policymakers should do with respect to incorporating soft skills into the school curriculum given the number of unanswered critical questions about how to proceed; and, third, to spur the organizations and individuals that are at the forefront of the movement to increase their interest and investment in the many unknowns of soft skills reform.

Defining the domain

The student dispositions, skills, traits, and abilities that are this report’s subject matter have been variously labeled as: soft skills, emotional intelligence, social and emotional learning, personal qualities, character, virtue, non-cognitive skills, 21st century skills, and so on. The topic spans vastly disparate categories of student behavior from easily observed actions such as completing homework, to abstract dispositions and ways of thinking such as optimism, grit, social awareness, and a growth mindset. The complexities and challenges for schools and educators of including such disparate behaviors, thoughts, and dispositions into the overarching grab bag of soft skills are large.

Schools that try to do everything are likely to accomplish nothing well. Thus, the first challenge for soft skills education reform is a coherent answer to the question: What are we talking about and trying to influence? 

Existing approaches to handling the dispersion, abstraction, and lack of coherence of a miscellany of soft skills largely take the form of creating subcategories that offer greater similarities among their members than the overall collection. For instance, Stecher and Hamilton distinguish between interpersonal competences such as the ability to work with others vs. intrapersonal skills such as having a growth mindset (believing that your abilities can be developed through hard work).[iv] The CASEL identifies five core competences: self-awareness, social awareness, self-management, responsible decision making, and relationship skills.[v] The authors of a Chicago Consortium report offer a different five categories: academic behaviors, academic perseverance, academic mindsets, learning strategies, and social skills.[vi] 

A leading researcher in the field, Angela Duckworth, and her colleague, Daniel Yeager, conclude that “the debate over the optimal name for this broad category of personal qualities obscures substantial agreement about the specific attributes worth measuring”[vii]—in Shakespearean terms, a rose by any other name would smell as sweet.

Duckworth and Yeager are correct that the issue is not the particular name that is used to refer to what researchers and advocates have substantially agreed should be measured. If that is all there was to it, we could take inspiration from The Artist Formerly Known as Prince and get the major players to adopt an unpronounceable symbol to refer to the domain. The important question, an affirmative answer to which Duckworth and Yeager implicitly assume, is whether the specific personal attributes that have been substantially agreed to as worth measuring are what schools should be measuring and held accountable for.

Given the number of unanswered questions about efforts to inculcate and measure soft skills in schools, a topic which I flesh out in the remainder of this report, it is premature and unhelpful for educators to define a school’s mission, select its curriculum and programs, measure its success, and be held accountable for something as amorphous as the various synonyms for soft skills. The “substantial agreement” among academic researchers about what should be measured is, at this point in time, unwarranted.

Distinguishing traits from behaviors

There is a strong relationship, both conceptually and empirically, between present approaches to the definition and measurement of soft skills and efforts in psychology going back almost 100 years to understand human personality. The earlier history of the taxonomic effort in the psychology of personality bears a striking resemblance to present day efforts to catalog soft skills. Just as soft skills are addressed now from a variety of perspectives and at different levels of abstraction, so too was personality addressed then.

The taxonomy of personality traits began to come together with the mid-century efforts by Raymond Cattell to isolate unique personality factors (he claimed to have found 16) using a statistical technique called factor analysis. Progress along this empirical pathway accelerated dramatically in sophistication and replicability of results as later researchers took advantage of advances in computing power and larger datasets.

Today, the field of personality psychology has converged empirically on the so-called Big Five personality traits: openness to experience, conscientiousness, extraversion, agreeableness, and neuroticism. Each is dimensional, i.e., a person can be high or low or somewhere in between on each of the traits. Each is statistically unique, i.e., an individual’s placement on the trait of agreeableness does not predict well where that person will land on conscientiousness. Each is typically measured using self-report questionnaires. And each refers not to specific behaviors in particular situations but to relative stable dispositions to respond in similar ways across a broad range of circumstances, i.e., there is no specification of what a person high on agreeableness will do or should do to be agreeable at school or in interactions with her peers or family members, only that she will find a way and be better at it than most.

Soft skills lack the century of empirical development that led to convergence on the Big Five personality traits, so the field is presently a Tower of Babel when it comes to constructs and measures. It is only in that regard that the field of personality traits and the field of soft skills differ. They share almost everything else, including the approach to measurement through questionnaires, the intent to capture broad patterns of behavior, and the goal of identifying individual differences that are predictive of later outcomes.

There is another, even more important, reason to link the domain of soft skills to the study of personality traits: they overlap substantially in coverage. For example, when the Chicago Consortium describes the social skills component of their model as “such interpersonal qualities as cooperation, assertion, responsibility, and empathy,”[viii] they are describing components of three Big Five personality traits. “Cooperation” and “Empathy” in the Consortium definition of social skills are defining characteristics of the Big Five trait of Agreeableness whereas the Consortium interpersonal quality of “Assertion” is one of the defining characteristics of the Big Five trait of Extraversion. Finally, the Consortium social quality of “Responsibility” is found in the Big Five trait of Conscientiousness.

The overlap is more than conceptual. In one of the largest studies of soft skills in school settings, the researchers used a Big Five self-report questionnaire as the basis for their measure of conscientiousness.[ix]

There is a significant challenge for soft skills reform if the focus of intervention and measurement is on broad trait-like patterns of behavior such as responsibility, cooperation and empathy. The problem is that the role of genetics and neurobiology looms large compared to the agency of any particular cultural institution, including schools.

As a case in point, the Big Five personality traits are highly heritable. In other words, behavioral differences among individuals in the qualities of openness to experience, conscientiousness, extraversion, agreeableness, and neuroticism are due in large part to genetic differences.

In behavioral genetics, the genetic contribution to individual differences in behavior is estimated by comparing the behavioral similarity across varying environments of people with known genetic similarities. The intuitively easiest case to understand is the comparison of the behavioral similarity of identical twins, reared together and apart.

Researchers find strong behavioral similarities in identical twins separated at birth and reared apart by different families. In fact, such twins are almost as similar in their behaviors and personalities as identical twins reared together. Consider one of many similar stories from media accounts: Paula Bernstein and Elyse Schein, identical twins adopted by different parents ended up living very similar lives: both became writers. As one of the twins remarked after getting to know her twin for the first time at the age of 35: “It’s not just our taste in music or books; it goes beyond that. In her, I see the same basic personality…”[x]

Several independent studies of twins converge on the estimate that a bit more than 40 percent of the variance in the Big Five Personality traits is due to genes whereas only 7 percent is due to the environment that is shared by twins raised together, i.e., home and school. For conscientiousness, the component of the Big Five that was used directly as a student self-report measure in a study of soft skills in the schools of Boston, the estimate of heritability from the four most recent studies is 49 percent.[xi]

School-level impacts on personality traits and trait-like soft skills are not incompatible with a 50 percent heritability of personality traits. In that regard, it is worth keeping in mind that we have lots of evidence that schools affect academic outcomes even though the heritability of scores on standardized tests of achievement at the end of school is, in some models, even higher than it is for the Big Five personality traits.[xii] Also, we know that the quality of the teacher and classroom to which kindergarteners are assigned affects teacher ratings of those students’ social skills years later and their earnings in adulthood.[xiii] But, is it reasonable to think that the school environment is more important than the family in shaping individual differences in broad behavior patterns such as cooperation and social awareness? I think not, although this is an empirical question.

In one of the most revealing studies to date on this issue, KIPP charter schools, which have the development of student character as one of their mission pillars, had an impact on only one of a wide range of student self-report measures of soft skills (collaboration/cooperation with other students), whereas these schools had significant impacts on academic achievement. The math and reading results and some of the soft skills results are presented in the following figure.


figure 1

The implicit theory of action for soft skills education reform is that, first, student self-perceptions of their self-control, grit, confidence (self-efficacy), and so forth are causally related to their achievement in school and their later life success; second, that schools can affect these self-perceptions through curriculum, school climate, and focused training; and, third, that the school impact on soft skills leads to improved student outcomes in other domains, including achievement.

The KIPP results illustrated in the figure above are not encouraging for this theory of action. KIPP is strongly committed and seriously invested in improving student character, which they view as important in its own right and a critical pathway to student success. The large and methodologically rigorous study on which the figure is based found that the only character/soft skill measure on which KIPP had an impact was student self-report of the extent of collaboration with other students. On everything else, including survey variables not included in the figure, KIPP students scored no better than identical students who wanted to attend the same KIPP middle schools but lost their lottery for admission. However, KIPP had a strong impact on the academic test scores of lottery winners. Thus, the route to better achievement in these KIPP middle schools did not flow through the enhancement of students’ soft skills and character. Or, if it did, none of the measures that were used in the study, which include those that soft-skills advocates cherish, picked up the changes in students’ soft skills that KIPP produced.

The evidence I have described and the larger body of research from which it is drawn suggests that there is a relatively low ceiling on the extent to which schools can affect individual differences in the broad patterns of behavior that are measured through self-report student surveys and conceptualized as soft skills. This does not mean that schools cannot affect the set point for perceptions of students about such things as their efficacy and effort. Although there is little evidence of that in the KIPP study, another rigorous study found that charter schools in Boston actually lowered their students’ self-ratings of soft skills while raising their test scores.[xiv]   

The heritability evidence I have noted means that it is very difficult for schools to change the relative ranking of students within a school on self-ratings of trait-like personal attributes such as conscientiousness. The evidence from the KIPP study and elsewhere suggests, further, that we know little empirically about how schools can exercise influence in ways that enhance soft skills and thereby improve academic achievement and life outcomes for students.

The need for specificity in what should be taught and learned

Consider the following definitions of “key competences” within the broad domain of social and emotional learning that are available from leading advocates of soft skills reform: social awareness: “The ability to take the perspective of and empathize with others from diverse backgrounds and cultures…”; responsible decision making: “The ability to make constructive and respectful choices about personal behavior and social interactions…”; soft skills: “personality traits, goals, motivations, and preferences that are valued in the labor market, in school, and in many other domains”; academic behaviors: “those behaviors commonly associated with being a ‘good student.’”

Contrast these abstractions with the specificity one finds when pivoting to the hard skills of reading and mathematics: For example, a Common Core State Standard for literacy for fourth graders is that students will “use combined knowledge of all letter-sound correspondences, syllabication patterns, and morphology (i.e., roots and affixes) to read accurately unfamiliar multisyllabic words…” A parallel Common Core standard for math for fourth graders is that they will “fluently add and subtract multi-digit whole numbers using the standard algorithm.” The clear force of such standards is that the curriculum that is delivered to fourth grade students has to include instruction and assessment on phonics and whole number arithmetic, and ideally leads to basic competency for all fourth graders in those skills.

Within the domain of soft skills there is nothing remotely close to this level of specificity in terms of what needs to be learned and when. We can probably agree, for example, that it is desirable for children and youth to engage in “responsible decision making.” But what should that look like for a second grader vs. a twelfth grader?

Existing efforts to merge Common Core academic standards with soft skills don’t get much beyond pointing out that there are opportunities to enhance soft skills while teaching the core academic subjects, i.e.,

The mathematical practices standards call for perseverance in solving problems, which supports the development of self-direction and productivity. In modeling, statistics and probability, for example, students can work in collaborative, project-based units in ways that emphasize Life and Career Skills.[xv]

Advocates for the inclusion of soft skills in the curriculum need to be a lot less nebulous. Without specificity at the level of what students need to learn and examples of how to teach it, there is no clear path to the development of curriculum and instructional practices, teacher training, or meaningful assessment and accountability.

Measurement and accountability

Reflecting its focus on broad patterns of thought and behavior and an academic grounding that is related to personality psychology, the soft skills movement has developed three types of measures of individual differences in soft skills: self-report questionnaires by students, teacher questionnaires about their students, and short performance measures, i.e., the famous Mischel Marshmallow Test assessing delay of gratification in which children are asked to choose between eating one marshmallow in front of them or holding out for a promised two—how long a child waits is the measure.[xvi]

In a recent report in Evidence Speaks, Martin West provides an informative account of some of the psychometric characteristics of the four self-report questionnaires being used in the CORE Districts in California—districts that are in the lead nationally in measuring students’ soft skills and holding individual schools accountable for how their students score.[xvii] In brief, West reports that four measures of student soft skills administered to middle school students (self-management, social awareness, self-efficacy, and growth mindset) have respectable levels of internal reliability, i.e., answers by individuals to questions within each scale correlate highly with each other. Examining school averages, the soft skills measures are positively correlated with math and ELA scores and negatively correlated with student suspensions and absences. Based on a dataset of 240 schools, the correlations between the four soft skills measures and math and ELA state test scores range from 0.33 to 0.69. Correlations with suspensions and absences are somewhat lower although still statistically significant. West concludes that his findings “provide a broadly encouraging view of the potential for self-reports of social-emotional skills as an input into [the CORE Districts] system for evaluating school performance.”

There are many questions that need to be answered and assumptions that need to be addressed before districts should be encouraged by such findings to give soft skills assessments to students and use the results to hold schools accountable, including:

  • What are the appropriate indicators of the validity of self-report measures of soft skills? It isn’t obvious conceptually that we should expect or want high cross-sectional correlations between soft skills and academic test scores in middle school. James Heckman, and others who have taken his lead, hold that interventions that impact soft skills such as pre-K play out later in life rather than in better test scores in school.[xviii] For example, Chetty et al. report that the differences between good vs. bad kindergarten classrooms can be seen in measures of student soft skills in middle school but not in academic test scores, and that it is these soft skills that lead to higher wages in adulthood for people who previously had a good kindergarten teacher.[xix] Further, the KIPP study previously referenced demonstrates a substantial independence between achievement gains and soft skills as measured through contemporaneous student self-reports. A study by West and colleagues indicates a negative relationship between achievement and soft skills impacts.[xx] At this point in time, we don’t know what level of correlation between a measure of self-efficacy and a measure of academic achievement is desirable or what any level of such correlations means with respect to the validity of the measure of self-efficacy.
    • Where is value-added? Data showing that schools vary in their mean scores on measures of soft skills can no more reasonably be used to hold those schools accountable than data showing that those same schools vary in their mean scores on measures of student vocabulary. In both cases family background and the selection of students into schools is presumptively more likely to be responsible for the measured differences than anything the school has or has not done for which it can to held to account. An accountability system based on test scores on measures of soft skills would need to capture school-level and grade-level gains on those measures so that schools are being evaluated on the changes they induce in students’ soft skills, i.e., their value-added.
    • Where is the teacher? One of the most powerful insights to emerge from the last 15 years of education research is the disproportionate role of the teacher and classroom relative to the school or district (or most anything else other than the family) in student outcomes.[xxi] Yet nearly all the attention among advocates of soft skills reform is at the district and school levels. In the CORE Districts, for example, it is schools that will be held accountable for social and emotional outcomes. It is likely that what teachers do in the classroom with respect to soft skills, just as for traditional academic skills, will be where most of impacts on students are to be found.[xxii] Further, the low-hanging fruit for improving soft skills will be found in interventions intended to improve the practices of or to remove from the profession those teachers who are negatively impacting students’ soft skills. We need classroom-level measures of soft skills to bring focus on where the education system is having the greatest impact for good or ill.
    • Why use proxy measures of valued outcomes when direct measures are available? West demonstrates that scores on soft skills questionnaires correlate negatively with student suspensions and absences.[xxiii] Which should a district, school, or teacher care most about, a student’s score on a measure of growth mindset or a student’s behavior that results in a suspension? Which should be a priority, an intervention that increases a student’s self-rating of conscientiousness or an intervention that increases timely completion of homework? Standardized test scores are a valuable and needed index of how much students are learning in the core academic disciplines. The test questions are close proxies for what students are expected to learn in their studies, the predictive validity of the assessments has been demonstrated repeatedly, and practical alternatives are not available. In contrast, a large portion of the domain of soft skills is directly observable by teachers and generates administrative records. To the credit of the CORE Districts, they collect and use the direct measures of suspensions and absences in their accountability system.

    We need more such measures. At the classroom level, more detailed report cards that include teacher ratings of such things as student effort are promising.[xxiv] Of relevan