The value of qualitative data for advancing equity in policy

Introduction

In response to President Biden’s Executive Order calling for a “whole-of-government equity agenda,” the Office of Management and Budget (OMB) released a report in July that provides guidance to federal agencies about what methods they can use to assess the extent to which their work advances racial equity and supports underserved communities in the U.S. The report is ground-breaking in its recognition of “administrative burden” and lack of stakeholder engagement as factors that exacerbate inequality, such as when citizens interact with government technology and eligibility processes. Yet, the OMB report falls short of explicitly recommending the use of qualitative methods (or a combination of qualitative and quantitative investigations, also known as “mixed methods”) to assess equity. This is a missed opportunity as qualitative data can reveal nuances of experiences that quantitative analysis alone cannot.

This piece articulates the value of bringing qualitative methods more deeply into policy research and practice in the U.S. We argue that a mixed methods approach, and especially a deeper appreciation of what qualitative research can contribute, is critical for capturing and validating the wide-ranging experiences of Americans. Inadequately incorporating qualitative approaches into assessing equity in U.S. institutions and policies will make it harder to understand how structural inequities impact Americans in varying ways and to take meaningful steps to address them.

The difference between quantitative and qualitative research

Quantitative research is the dominant methodological paradigm in policy debates and social policy research. Rooted in the philosophical traditions of logical positivism, quantitative research focuses on explanation and causation, typically using statistics and modeling to assess the strength, significance, and generalizability of a given relationship. For example, the research question, “Does wearing masks reduce COVID-19 transmission?” is aligned with a quantitative research paradigm. Quantitative research methods often strive to isolate the causal effects of the specific variables the researcher is interested in observing (e.g., mask wearing, COVID-19 transmission) through the use of experimental and quasi-experimental methodologies that can divide people into “treatment” and “control” groups, and the use statistical techniques to examine whether these groups are statistically different from each other. Quantitative researchers try to maximize the sample size to identify general patterns and trends with a reasonable level of certainty. Typically, the researcher chooses a set of research concepts and categories to measure and test from the outset, informed by the existing literature, exploratory analysis, and available data.

A mixed methods approach is critical for capturing and validating the wide-ranging experiences of Americans.

In contrast, a qualitative research paradigm centers everyday human experiences and understandings of the world. It is rooted in meaning-making and shines in its ability to capture the richness and depth of the research context. Because of these goals, qualitative research is valuable for situating and interpreting findings in a specific context or capturing how a given issue may be understood from different positions or angles. For example, the research question, “How do gender roles and perceptions of social group identity in the U.S. South shape mask-wearing behaviors?” is aligned with a qualitative research paradigm. The process of conducting and analyzing qualitative research – which can include focus groups, interviews, discourse analysis, or observation – is agile and iterative, which allows the researcher to probe deeper into a given theme or social process. Researchers and administrators can choose from a wide range of qualitative methods along the spectrum of depth and breadth based on the common practices in their discipline, available resources, and the research questions and goals. Approaches can range from ethnographic studies that involve immersing oneself in a research context for more than a year to holding a set of one-hour focus groups.

There has been a general movement in U.S. policy research towards establishing a hierarchical categorization of “tiers” of evidence for policy studies, with experimental quantitative methods such as randomized controlled trials (RCTs) positioned as the “gold standard” for making evidence-based policy decisions. Critics have highlighted the limitations of centering experimental methods alone (especially in social policy) and the consensus has started to shift more towards mixed methods.

Benefits of bringing qualitative research into policy

Rigorous qualitative data collection and analysis is not merely a way to bring life to quantitative evidence about policies or programs (although it often does that powerfully). Qualitative research allows the researcher to gather rich contextual insights into people’s lived experiences of policies, programs, and power dynamics. When it comes to advancing equity in policy, qualitative research can be a useful approach to understand how and why a given program or intervention may or may not work as intended (and for whom) and how to improve it from the perspective of a specific group of stakeholders. These insights present policy researchers and administrators the opportunity to steer policies and programs in a direction that is rooted in how people experience a program or service in practice and its influence on their day-to-day life and world view.

For example, a small business owner who has to fill out long application forms with confusing questions and reporting requirements to access a grant or loan may develop a perception that the program is more hassle than it is worth, which may discourage them to apply and give an advantage to larger companies that have the resources and time to comply with the requirements. Qualitative research, such as interviews or focus groups, could have been used to gather input from small businesses to design a process that is accessible and feasible for them, but that also helps meet the administrative needs of the program.

Qualitative analysis also allows policy researchers and administrators to embrace, rather than flatten, differing perspectives on the same concept, such as “middle class” or “empowerment.”

Qualitative analysis also allows policy researchers and administrators to embrace, rather than flatten, differing perspectives on the same concept, such as “middle class” or “empowerment.” This is critical in the formulation of equitable policy and practices, because people exist at the intersection of multiple identity markers and can experience and interpret the same policy in vastly different ways. Researchers and administrators can use qualitative methods to test and validate a priori conceptual categories (e.g., key concepts in the research questions), identify new variables (e.g., quantitative metrics included in a statistical model), and generate fresh hypotheses. This helps to ensure that the research is grounded in the ways that people understand their own realities and priorities, rather than only the realities and priorities of the researcher. Qualitative research then allows policy makers and administrators to target interventions for specific populations and, in the case of equity analysis in policy, to develop interventions that address dynamics that are important drivers of structural inequality.¹

For example, some economists have framed the debate about who should qualify for student loan debt cancellation in terms of family income and post-college earnings, making the argument that targeting forgiveness in this way would “benefit families that are poorer, more disadvantaged, and more likely to be Black and Hispanic.” However, the choice to propose a means test based on family income or post-college earnings rather than wealth fails to acknowledge the important role that generational wealth plays in a borrower’s economic security and mobility. The history of legally sanctioned practices such as redlining and blockbusting created unequal access to generational wealth in the U.S. on the basis of race, which kept Black Americans from building equivalent wealth through homeownership. With disproportionately low levels of family wealth, Black borrowers remain at a structural disadvantage for achieving home ownership, starting a business, or other wealth-generating activities even if they have high post-college earnings due to the lack of upfront capital to obtain financing. When researchers and administrators fail to consider how systems associated with one’s gender, race, and class shape differential impacts of such proposals, they risk reproducing structural inequality rather than addressing it. In this case, structural inequality in wealth is a key concept to focus on when designing a student loan forgiveness policy which advances equity. Qualitative research that engages student borrowers would be uniquely suited to reveal the differential impacts of student loan debt on specific populations of borrowers as it would enable the researcher to gather rich information about what factors or combinations of factors shape their economic mobility and well-being.

Case study: using qualitative data to design policy for the middle class

The Future of the Middle Class Initiative (FMCi) at Brookings launched in 2018 with the goal of better understanding how to improve the quality of life of America’s middle class and increase access to it. From the beginning, FMCi relied on nationally representative quantitative data on how the middle 60% of the income distribution (how the Initiative defines the middle class) has been doing over time. For example, the team analyzed Congressional Budget Office (CBO) data to reveal that middle-class income growth is lagging behind income growth for rich and poor people, and Census data² to demonstrate that middle class marriage rates were declining. Money and relationships are two factors understood to contribute to well-being; however, the available quantitative data on these, and other key pillars of well-being, in our work did not allow the research team to understand the contextual factors that shaped how middle-class Americans perceived their own well-being, examine power dynamics within the middle class, or appreciate the wide range of experiences of middle-class Americans.

To address these gaps, the team added a qualitative research component, called the American Middle Class Hopes and Anxieties Study. This study, stratified by race and gender, allowed the team to gain a richer understanding of middle-class experiences. In focus groups, Black and Hispanic participants reported frequent experiences of discrimination at work and racist interactions with police or, as one Black man called it, the “injustice system.” These participants shared that their widespread and repeated experiences of racism shaped their overall well-being. If researchers or administrators develop policy interventions that fail to address the pervasive negative effects of racism, the qualitative evidence suggests that they are unlikely to see meaningful improvements in well-being for Black and Hispanic middle-class Americans. The qualitative evidence also provided information about how to prioritize interventions, such as increasing the policy focus on policing and the criminal justice system.

These same data were used to expand our understanding of the middle-class time squeeze. Previously, our colleagues’ quantitative analysis centered dual-earner middle-class families with children, based on the assumption that these households face a particularly tight time squeeze due to the rise in women’s labor force participation and the prevalence of dual-earner married households with children in the middle-class. Follow-up mixed methods analysis demonstrated that a much wider range of middle-class Americans experienced a time squeeze – including Americans from different occupations, unmarried individuals, and a wide range of Americans of different races, ages, and genders. There was an overarching pattern among participants indicating that most middle-class Americans in our focus groups felt pressed for time to do the things that bring their lives meaning – spending time with their families, time on self-care, or time in leisure. Many participants described their work as the primary driver of how they structure and think about their time. Our qualitative findings suggest that the middle-class time squeeze may have deeper social impacts than is widely understood, and therefore it allowed the research team to generate new hypotheses about the importance of workplace culture, norms, labor protections, and a worker’s sense of autonomy in shaping well-being.

Understanding quality and rigor in qualitative research

When researchers or organizations are new to qualitative research, they often run into major stumbling blocks and are hesitant to adopt qualitative or mixed methods deeply into their work. At best, a novice may see qualitative insights as useful for narratives that can help reach wide audiences because it helps personalize numbers and statistics (akin to journalism). Yet they often express concerns that qualitative research is “unscientific” and susceptible to bias. However, it is problematic to simply copy and paste a quantitative framework for assessing the quality and rigor of a given qualitative research framework. Qualitative research paradigms are rooted in different philosophical traditions, with different goals and priorities. As such, the norms for achieving generalizability, validity, and quality in qualitative research are different.

For example, a common critique of qualitative research is that sample sizes tend to be too low to maintain rigor in qualitative research. However, as noted earlier, qualitative research prioritizes meaning making and understanding social processes. Qualitative researchers therefore place more priority on maximizing the depth of the data collected than they do on the size of the sample, per se. The aim is to gather information from different perspectives (i.e., triangulation of multiple qualitative methods, a wide range of informants) in a specific context until they reach saturation – the point at which further research fails to produce new themes relevant to the research questions. While there is no set minimum sample size, most researchers aim for a minimum sample of 15 to 20. If there is a lot of variation across informants or if the scope of questions is very broad, more research will be necessary to reach saturation.

There are other norms in qualitative research that quantitative researchers may not expect. For example, although qualitative researchers are also expected to be transparent about methodological choices and limitations, objectivity is neither assumed, nor is it typically viewed as possible to fully achieve. A recent debate in response to a piece that claimed that objectivity in and of itself is harmful highlights the sensitivity around the topic of objectivity and researcher bias, and what it means for the ways that rigor is approached in different fields of research. Although we do not view objectivity as harmful (and in fact, it is important to strive towards it in order to reduce researcher bias), the fact that many researchers or administrators believe that their work is inherently objective without critically examining their own biases and a priori assumptions can be harmful, inaccurate, and can lead to policy interventions that have unintended effects. For example, the American Middle Class Hopes and Anxieties Study work discussed above helped the research team to unpack race-gender dynamics within the middle class that many research team members were not previously attentive to, in part because it was not a salient part of their own experience. Qualitative researchers often start their research from the assumption that all knowledge is partial and situated, meaning that any individual researcher approaches their work from the position of their own privileges, biases, and lived experiences.

Like quantitative researchers, qualitative researchers are expected to be attentive to ethical obligations in research. In a qualitative research setting ethical dilemmas tend to occur due direct engagement with research participants on detailed and sometimes sensitive information. In qualitative research, it is especially important to consider when the research involves uneven power dynamics between the research team and research participants, sensitive or potentially dangerous research topics, vulnerable populations, or the potential to disclose identifying information that could lead to economic, social, or physical harm for the research participant or their community. Critically examining the power relations between the researcher and research participants and how the researcher shapes the research context and findings is typically an important element of producing quality work.

Finally, while quantitative researchers often strive to reduce complexity in pursuit of understanding causal relationships (e.g., by controlling for several variables to isolate the effects of a variable of interest), qualitative researchers prefer to dive into complexity to understand the social processes more comprehensively. This is one of the reasons why combining qualitative and quantitative methods in mixed methods projects can generate a deeper understanding and open up new areas of inquiry that neither method could reach alone.

Combining qualitative and quantitative methods in mixed methods projects can generate a deeper understanding and open up new areas of inquiry that neither method could reach alone.

Towards a mixed-methods approach to policy research

As powerful as rigorous quantitative research may appear on the surface, it can only ever offer a limited perspective on equity. Qualitative research can capture complex dynamics or variables that would otherwise be undetectable, help researchers and practitioners gain critical insights into their own blind spots and biases, and explore the ‘how’s and ‘why’s of interventions, program outcomes, and human experiences.

The Biden Administration has called to center equity in policy in order to reduce systematic inequality in U.S. laws, policies, and institutions. To fulfill the executive order, agency staff and policy researchers should require mixed-methods approaches in any equity assessment to capture policy-relevant dynamics and perspectives that otherwise would be excluded. There are existing resources, handbooks, and toolkits that practitioners can draw from, as well as resources from applied fields such as human-centered design. Bringing in qualitative methods (e.g., interviews, focus groups) also requires additional expertise and adequate budgets to support recruitment, site visits, collection and analysis, incentives, and ethical review processes.

A deadly pandemic, national protests, and a global climate crisis have all had disparate impacts on the most marginalized members of our society. Policymakers, researchers, and administrators must make a concerted effort to better understand lived experiences, the social processes that shape them, and how that manifests in U.S. policy and programs – especially if the goal is to address systemic, complex forms of inequity in ways that have a meaningful impact on the lives of all American people.

The Brookings Institution is financed through the support of a diverse array of foundations, corporations, governments, individuals, as well as an endowment.  A list of donors can be found in our annual reports published online here. The findings, interpretations, and conclusions in this report are solely those of its author(s) and are not influenced by any donation.