Agency learning agendas and regulatory research

As federal agencies scramble to respond to COVID-19, there’s another initiative rolling forward that you probably haven’t heard of, one that was built by data wonks and will influence regulation in ways we’re just now beginning to understand. As part of a new legal requirement, agencies are writing learning agendas to organize the way they approach research, including regulatory research.

Learning agendas will allow agencies to coordinate their research plans internally and externally, including with outside researchers who can help answer agency questions. But learning agendas, like many public processes, are subject to the pernicious influence of rent-seeking. As agencies consider which questions to add to their learning agendas, they should be wary of those who try to shape agency questions for their own gain.

Especially in these early days of learning agenda implementation, agencies should tread carefully with regulatory questions. The questions contained in Executive Order 12,866, the long-standing directive that guides regulatory development, offer a steady path forward. These questions are time-tested and provide an alternative to the questions that various individual interests might prefer. Learning agendas could help bring these questions to the front of regulatory processes, which is where they belong.

The Evidence Act

The Foundations for Evidence-Based Policymaking Act (Evidence Act) is an impressive bundle of good government reforms that grew out of several years of painstaking bipartisan and nonpartisan work. In 2015, Speaker Paul Ryan (R-WI) and Senator Patty Murray (D-WA) jointly introduced the bill that would go on to create the U.S. Commission on Evidence-Based Policymaking in 2016. The Commission, with members drawn from academia, government, and the private sector, offered 22 recommendations in its report in 2017. Eleven of these recommendations were reflected in the Evidence Act, which was also jointly introduced by Ryan and Murray and signed into law in 2019.

Although all four titles of the Evidence Act have implications for regulatory agencies, Title I is the most directly relevant.[1] It inserts new requirements into existing agency planning processes. That is a dry-sounding but ultimately quite savvy way to increase the odds of smooth implementation. First, as part of its usual strategic planning process, each agency must now prepare a learning agenda to identify and address policy questions relevant to the agency’s work. Second, as part of its usual annual performance plan, the agency must now include an evaluation plan describing each significant evaluation that the agency plans for the coming year. While hooking these new actions onto existing planning processes does not guarantee complete success, it takes advantage of existing agency workflows rather than starting from scratch.

Title I applies to the 24 agencies subject to the Chief Financial Officer Act, which includes big regulatory agencies like the Environmental Protection Agency and the Departments of Transportation, Health and Human Services, and Labor.[2] While the Evidence Act is not aimed exclusively at regulations, it intersects with them because the Act’s learning agendas and evaluation plans apply to “policy questions relevant to the programs, policies, and regulations of the agency.”

As is probably clear from this description, this law is focused on the inner workings of government. Unlike regulatory reform bills that would alter aspects of the notice and comment rulemaking process, the Evidence Act makes changes to the way the government generates and integrates information into its various policy processes including regulation.

Evaluation plans have the potential to spur long-sought, meaningful retrospective review of regulations. They can help agencies learn from their prior regulations, something that, right now, does not happen in a systematic way. Learning agendas set the stage for these evaluation plans, and they are due soon.

The Evidence Act makes changes to the way the government generates and integrates information into its various policy processes including regulation.

Learning agendas

An agency learning agenda is what it sounds like. It is a multi-year set of research questions that an agency focuses on going forward. The questions can be a blend of short-term (e.g., how well do this year’s grant recipients understand their compliance obligations?), long-term (e.g., does a new employee recruitment strategy lead to less turnover?), strategic (e.g., does partnership with local non-profits lead to better beneficiary uptake?), and operational (e.g., do email or phone reminders help avoid missed student loan payments?). The options are almost endless. These are questions that agencies are likely trying to answer whether they are written down in an agenda or not.

So why write them down? Without a learning agenda, different agency programs likely pursue different questions, possibly overlapping, in an uncoordinated or loosely coordinated manner. Funding and other resources may or may not be allocated based on priorities. That status quo isn’t necessarily so bad until you consider the alternative: a learning agenda as a means to prioritize research questions and align resources, including outside parties, to answer them.

It will take considerable effort for agencies to wrangle their research questions into manageable buckets and prioritize them. Apart from the internal coordination benefits, which may or may not be enough to justify this work, the learning agenda opens up the possibility of coordination across different agencies with related programs and with others outside the federal government. This could include, for example, state, local, and tribal governments, non-profits, universities, think tanks, and foreign researchers and governments.

Think of it this way: the agency has questions, probably many more questions than it can answer alone given resource constraints, and expertise to answer them is scattered across different institutions. While some research already occurs under federal grants, the learning agenda could enable a less formal mode of collaboration between academia and government. It promotes opportunities for scholarship with greater policy relevance and policy that’s better informed by research. How this will all work, exactly, is still in flux. Fortunately, the process is designed to work in multi-year cycles, snapping right into the already-institutionalized strategic planning timeline, so progress does not require perfection.

Some agencies were producing learning agendas or similar documents before the Evidence Act was enacted, so they have a head start. The Department of Housing and Urban Development offers one example of an agency-wide learning agenda, the Small Business Administration another, and the Department of Labor’s chief evaluation office coordinates learning agendas across the department’s different sub-agencies.

Other agencies are on a glide-path, with the Office of Management and Budget (OMB) keeping close watch. Six months after the Evidence Act was signed into law, OMB issued its first tranche of guidance to agencies. The guidance sets out a timeline, with agencies publishing their learning agendas by February 2022 along with their next strategic plans. Along the way, agencies are expected to submit several iterations of their learning agendas to OMB for review. The next deadline is September 2020, synched to when agencies submit their budget cycle submissions.

The promise and peril of learning agendas

Directing agencies to use “evidence” for policy decisions is about as anodyne as it gets. As Oren Cass wrote in his blistering critique Policy-Based Evidence Making: “Who, after all, could be against evidence?” But one does not have to be “against” evidence to have some concerns about how policies like these are implemented.

A key concern is that “evidence” is only as good as the way it was built; biased research questions and methods lead to biased results. And, even strong evidence can be twisted to suit the ideological commitments of policy-makers. Meanwhile, social science is still in the throes of its replication crisis, which shows one major limit of relying on even reasonably well-designed studies. But the solution is not, obviously, to abandon the use of data or evidence in policy-making. It is instead to be scrupulous about how that information is gathered and used.

A key concern is that “evidence” is only as good as the way it was built; biased research questions and methods lead to biased results.

Learning agendas are one domain in which agencies should exercise discipline, particularly with respect to public consultation. Agencies are required to consult with stakeholders as they develop their agendas, but they have flexibility with respect to how they do it. Possibilities include one-on-one consultation in meetings, phone calls, or over email; surveys; technical working groups; public meetings or listening sessions; opening a public docket for comments; and more. Agencies are not required to make their approach public, but they should.

OMB directs that stakeholder engagement should “make sure that the learning agenda addresses questions that are relevant, salient, and meaningful to those with direct interests in the agency’s functions.” While focusing on “stakeholders” is likely to draw in genuine expertise, it also invites rent-seekers to the table. The opportunity to influence an agency’s research agenda towards one’s own interests is likely too tempting to resist. Why not advocate for the study of an investment that will benefit oneself?

We do not have a very good screen for this behavior in any area of generalized government decision-making. To some extent, the public disclosure of learning agendas will mitigate the influence of rent-seekers, but agencies should always consider how stakeholders’ particularized interests may conflict with the public’s interest as a whole. This is very hard to do, in practice, especially as rent-seekers dust off their halos and wrap their arguments in high-minded rhetoric that they might even believe.

So what is an agency to do? Who, after all, could be against public consultation?

A good source of regulatory questions for learning agendas

As agencies get to work on their learning agendas, they will likely struggle with how to craft questions related to regulation. On one extreme, it might be tempting to include questions about what the agency should ultimately decide where it has legal discretion to so do, e.g., should eligibility for a program or permissible exposure limits be set at this or that level. These kinds of questions are generally not evidentiary questions but rather policy decisions that must be informed by data and science, law, and policy preferences. On the other extreme, overly specific technical questions that are decoupled from an agency’s regulatory authorities risk being unactionable.

So it is clear that the quality of the questions in a learning agenda will influence how useful it is to regulatory decision-making. It is also clear that rent-seekers will have incentives to infiltrate agency choices about learning agendas, through public consultation channels or others.

Fortunately, agencies do not need to reinvent the wheel, even as they work to produce new learning agendas. Executive Order 12,866, which has been in place since 1993, contains twelve principles of regulation that can be turned into learning agenda questions. The table below shows four examples.

Four examples: Translating the principles of regulation into learning agenda questions

If an agency is looking for good questions to ask about its regulations, and wants to navigate around rent-seeking behavior, Executive Order 12,866 shows them the way. These principles have been embraced by presidents of both parties through a period of intense political polarization. They also organize agency inquiry in a way that is probably more general than a specific interest might prefer. In these early days of learning agenda implementation, this might help mitigate rent-seeking.

In addition, a long-running problem with implementation of Executive Order 12,866 is that the policy questions it directs agencies to ask are sometimes answered at the end, after a policy decision has already been made. This is especially vexing timing if the agency is writing a rule that Congress expressly required them to write. This is similar to the critique above; the analysis or evidence comes into the process too late, after minds are made up.

While the Evidence Act does not fully resolve all of these issues, it does make some progress. The learning agenda creates an opportunity to pull regulatory questions towards the front of the policy development process. The risk of these questions being overrun by regulatory rent-seekers is at least somewhat mitigated by drawing on the principles of Executive Order 12,866 to form the questions. At the same time, learning agendas offer a chance for new and productive coordination both inside and outside government. Whether they’re worth it is perhaps a good question for a future learning agenda. 

The author did not receive financial support from any firm or person for this article or from any firm or person with a financial or political interest in this article. She is not currently an officer, director, or board member of any organization with a financial or political interest in this article.