GAMECHANGER: A case study of AI innovation at the Department of Defense

Executive summary

Taking advantage of rapid advances in artificial intelligence (AI) is an active and ongoing priority for the United States Department of Defense (DOD). While attention has largely focused on AI’s role in warfighting systems, the Pentagon’s vast business operations—finance, procurement, internet technology (IT), health care, supply chain management, and more—are equally critical. Indeed, the DOD is the country’s largest employer, provides health care to 9.4 million people, owns and/or manages more than 700,000 facilities worldwide, is fiduciary of almost 1 trillion taxpayer dollars (and growing), and is charged with buying and firing weapons in ways that are legal, ethical, and effective. Whether and how well the DOD uses AI to enhance the voluminous set of workflows required to dispatch these “back room” and “boardroom” duties, therefore, has implications that extend well beyond the battlefield.

This report is a retrospective examination of one such effort. It offers a snapshot in time, focusing on the development of GAMECHANGER, a large language model-based application. It was designed to enhance access to the large volume of government authorities, directives, and policies that govern the DOD’s administrative and military operations. GAMECHANGER was codeveloped in 2018 by the government in partnership with the private advanced technology company Booz Allen and implemented in 2020. It therefore predates the release of GPT-3, and the generative AI fervor triggered by its successor, ChatGPT.

The emergence of these tools does not degrade the utility of GAMECHANGER to DOD users, and it does not diminish the extent to which the application’s development represents real innovation at the DOD. The project was initiated at the urging of an entrepreneurial employee with the vision, conviction, and skill to develop a problem-solving application of AI. The DOD proved able and willing to adapt its standard practices for collaborating with private companies, found ways to make accommodations for the GAMECHANGER development team’s particular needs, provided dedicated funding from inception through deployment, and ultimately delivered a search tool that has accelerated the rate and enhanced the ease with which employees access and interact with the policies that guide the DOD’s activities.

A retrospective review of GAMECHANGER’s development also reveals practices and offers lessons that the department can usefully apply to its ongoing efforts to modernize, streamline, and innovate its enterprise technology. The GAMECHANGER development team encountered systemic barriers that the government can and should address, including overly rigid network security requirements, the lack of a dedicated DOD career path for technical experts, and leadership turnover that caused inefficiencies and ultimately cost the program its champion. Continuing to make accommodations for AI-enabled software development projects on an ad hoc, case-by-case basis is insufficient to meet the DOD’s medium- and long-term technological needs.

AI at the DOD

The United States Department of Defense has been engaged in the development and application of AI-enabled technologies since the 1940s. The extent of DOD AI investment and activity over those eight decades has fluctuated with the rate of technical progress in research institutes and in industry, with the budgeting priorities of presidential administrations, and with the nature and types of threats present in the international environment. These influences converged to create substantial change in the mid-2010s. Technical advances, including data availability and increases in computational power, together with the emergence of China as a geostrategic competitor, convinced high-level military officials that the DOD needed an intentional, systematic approach to incorporating AI into its operations.

Over the subsequent decade, the weight of attention devoted to monitoring the department’s progress on acquiring and adopting AI-enabled technologies has been determined by the extent to which and the ways in which AI is being integrated into warfighting platforms and technologies. These warfighting capabilities, however, do not emerge of their own; they are the output of the people and processes that constitute the DOD’s vast business enterprise operations: compliance, finance, procurement, IT, health care, supply chain management, talent management, property management, and more. Indeed, the DOD is the country’s largest employer, provides health care to 9.4 million people, owns and operates more than 700,000 facilities worldwide, is fiduciary of almost 1 trillion taxpayer dollars (and growing), and is charged with buying and firing weapons in ways that are legal, ethical, and effective.¹ Whether and how well the DOD uses AI to enhance the voluminous set of workflows required to dispatch these duties, therefore, has implications that extend well beyond the department’s ability to wage war.

This report provides one view into how the DOD is identifying the opportunities and addressing the challenges of integrating modern AI tools into its business enterprise operations. It describes the origins and outcomes of GAMECHANGER, a search engine built between 2018 and 2022 to “identify, consolidate, and automate the discovery and analysis of all applicable Statute, Executive Orders and Presidential Directives, Regulations, DoD Issuances, Military Department guidance, and other relevant USG policy” that define the roles, responsibilities, and activities of the department’s many programs, offices, and civilian and military employees.² GAMECHANGER, in other words, is a software tool DOD employees can use to search for and to access the policies—its own, and those of other government entities empowered to do so—that guide its conduct. It is, as one of its developers described it, a “Google for DOD policies.”³

The U.S. government and Booz Allen, a private advanced technology company, jointly built GAMECHANGER for use by the Department of Defense; as such, the team and the program that developed it have some unique attributes. Yet a number of elements of the GAMECHANGER story are also quite familiar: it begins with a broken workflow, has an entrepreneurial and obsessive founder, and features chance, happenstance, good luck, bureaucratic resistance, organizational pathology, internal disagreements, and external shocks.

The program traversed all these hazards and emerged as an example of true innovation. The Department of Defense proved able and willing to respond to an employee (the “founder”) with the vision, conviction, and skill to develop a problem-solving application of AI. The DOD adapted its standard practices for collaborating with private companies, found ways to make accommodations for the GAMECHANGER development team’s particular needs, provided dedicated funding from inception through deployment, and ultimately delivered a search tool that has accelerated the rate and enhanced the ease with which employees access and interact with government policies. The fact that these impediments required resolution, however, points to a need for the department to become far more able to develop, acquire, and adopt software. Continuing to make accommodations on an ad hoc, case-by-case basis is insufficient to meet the department’s medium- and long-term technological needs.

Origins

GAMECHANGER has its origins in frustration. Over multiple years and in different environments, GAMECHANGER’s founder had either observed or experienced many instances in which DOD action was impeded, distorted, or prevented by an inability to access, cross-reference, and reconcile its own authorities and directives. This slowed administrative functions—responsiveness to congressional oversight processes, for example—and it also had direct, operational implications. GAMECHANGER’s founder described being especially angered when planning for counterterrorism missions was negatively affected by a poor understanding of which policies were applicable and confusion about how to comply with them. Operators in the field had to ask compliance questions of lawyers who were working from a distance; the lawyers from a distance didn’t have the necessary context or ready access to all of the relevant facts, and so the decision cycle was slow and the results often insufficiently directive.

Repeated exposures to such exchanges produced a self-described obsession with the idea that if “people knew what policies and directives said and how they interacted with each other, we could fix a lot of things, across any number of issues.”⁴ Figuring out how to give people a tool, an application, with which they could quickly access policy and receive guidance about how to apply DOD policy thus “became a personal mission.” This developed into a vision of decision enablement; GAMECHANGER’s founder wanted field operators to be able to input mission details and descriptions of current conditions and have an AI-powered application analyze DOD policies and authorities and rapidly return an assessment of what actions were, and were not, permissible.

This idea was evolving at the same time the Department of Defense was beginning to work in earnest on digital modernization with the intent of expanding its use of AI-enabled autonomy for military operations, and advancing its overall approach to data capture, processing, and analysis. These efforts produced some publicly visible outcomes. In the warfighting domain, for example, the department began to experiment with more, different, and better uncrewed platforms, commonly called drones, in the air and on and in the ocean. It also initiated Project Maven, which used AI image recognition capabilities—so-called “computer vision” —to identify targets during kinetic operations.⁵ This emphasis on AI and automation also resulted in new programs and other organizational changes that would allow the department to take much better advantage of the massive volumes of data generated by, and potentially useful to, its business enterprise operations—to become, that is, a “data-driven organization.”

Consistent with this intent, the DOD launched the “Advancing Analytics,” or Advana program, in 2018. Advana’s purpose is to serve as a repository of data—a “data lake”—about the department’s activities. It collects and reconciles data from disparate authoritative systems and allows analysts to use programming languages and technical tools to analyze those data, and to create new datasets and visualizations from them that are useful to their work. It is available to all civilian, military, and contractor employees with a common access card, a form of identification issued and administered by the DOD.

Although initially designed to enable the DOD comptroller to track and report on the department’s financial activities, during the COVID-19 pandemic, analysts used Advana for a number of purposes beyond financial management. These uses demonstrated the value of giving employees access to DOD data, and so Advana’s mandate expanded to include data on a more comprehensive set of DOD activities, including, for example, logistics, personnel, equipment and asset readiness, acquisition, and medical operations.

It was also during 2018 that the DOD established a new office, the Joint Artificial Intelligence Center (JAIC), under the office of the chief information officer. The JAIC’s mandate was “to enhance the ability for DoD components to execute new AI initiatives, experiment, and learn within a common framework” with “the overarching goal of accelerating the delivery of AI-enabled capabilities, scaling the Department-wide impact of AI, and synchronizing DoD AI activities” to “swiftly introduce new capabilities and effectively experiment with new operational concepts in support of DoD’s warfighting missions and business functions.”

The person charged with directing the JAIC was U.S. Air Force Lieutenant General Jack Shanahan, whose prior assignment was as the head of Project Maven. GAMECHANGER’S founder had an acquaintance with Shanahan while he was leading Project Maven, and in late 2018, approached him with the idea of having the JAIC develop a tool that would solve the problems of policy inaccessibility and incomprehensibility that the founder knew were impeding DOD operations—a tool that the founder told Shanahan would “save the Department billions of dollars, and lives, and is congressionally mandated.” According to the founder, Shanahan and his deputy director were responsive to this argument not only because they understood the problems the founder was proposing to solve, having themselves “experienced it as a pain point,” but also because they understood the technical concept.⁶ Shanahan thus made the administrative maneuvers necessary to establish work on a policy tool prototype and designated it a flagship JAIC project. Between 2019 and 2024, the GAMECHANGER program received somewhere between $8 million and $15 million in funding.⁷

Development

GAMECHANGER’s founder was not initially a member of the JAIC, but instead the client—the person for whom and with whom the project team would design and build the AI-enabled tool. During much of the JAIC’s first year, it was heavily staffed with people on temporary assignment—employees who rotated in from other offices where they had roles that were not specific to the JAIC’s activities. This was true of the subset of people staffed to work with the founder on GAMECHANGER for about eight months, at which point the JAIC hired an experienced technical project manager from another national security agency. It was around this same time that Shanahan met the DOD’s chief data officer, Michael Conlin.⁸

Conlin had arrived at the DOD in July 2018 after working on digital transformation in the private sector, and was part of the internal DOD team that “conspired to create Advana.”⁹ Conlin’s staff included two members with advanced data science and data engineering skills, whom he had empowered to engage with the core Advana team to find and solve data problems of importance to the department.¹⁰ Both team members, for different reasons, identified the inability to work meaningfully with the full set of DOD policies, authorities, and directives as that problem. Shanahan, recognizing the overlap, connected GAMECHANGER’s founder to Conlin and his staff, and they began an active collaboration, becoming, in the founder’s words, “a little team.”¹¹

The team’s initial work produced a proof of concept in the form of computer code that identified interconnections across a set of policy documents. This was encouraging but not a cause for optimism about the larger intent. As described by one team member, these results were only “toy things”; the department’s technology and the restrictions imposed by its security policies prevented them from being developed further. These constraints meant that the team had no means of moving code from an individual computer to a shared network that would allow direct collaboration, and no access to the widely used, cost-free, open-source software development tools they would need to build a complete and capable application.¹² The team, understanding that these department-wide rules and regulations were unlikely to change, much less on any reasonable timeline, concluded that the only feasible way forward was to push the JAIC to engage a private company.

GAMECHANGER’s founder had a strong preference for which private company that should be. GAMECHANGER’s near-term goal was to produce a policy reconciliation tool, but the founder also wanted to retain the option of additional applications thereafter, including the possibility of a suite of capabilities that would “correlate policy with all other government data.” If the policy tool could connect to data on budget and spending, for example, users could inform decisions by running estimates of how policy changes would affect costs. This, to the founder, meant that GAMECHANGER needed to be able to run either as an application within or with access to Advana, and the department’s partner on Advana was Booz Allen. The Advana project had already demonstrated its ability to move applications into production quickly, and Booz Allen was willing to develop the application using publicly available, non-proprietary tools. The JAIC, therefore, was able to successfully negotiate the addition of the GAMECHANGER project to the existing Advana contract, with the condition that the members of the founder’s “little team” would retain leadership roles.

This meant that, in early 2020, Booz Allen assumed responsibility for the development of GAMECHANGER. Booz Allen would build and manage development on its own infrastructure and connect the application to the government’s network upon completion. There was some disagreement about whether the team could develop the tool successfully without direct access to DOD policy documents, which are held on government networks. The availability of large portions of these materials on open public systems solved this problem.

There were also initial frictions in the relationship concerning what government staff could and could not do in Booz Allen development environments. Members of the little team were insistent about participating in the policy application’s coding. This would require them to have direct access to Booz Allen technology and infrastructure, raising risk and liability concerns for the company. Both Booz Allen and the government ultimately made a number of non-trivial adjustments to their standard practices in order to make the relationship and the project work. These included tolerating compromises and arriving at agreements on hiring and staffing, code ownership rights, and the use of open-source software to develop GAMECHANGER. GAMECHANGER’s code base was also made open-source.

Open-source software

Most people are familiar with software—whether buying it, downloading it for free, or using it to complete everyday computing tasks. Today, it is more often called an application (or “app”) or a program. Less commonly, it is referred to as code—even though code is precisely what software is made of: a set of defined instructions, written in computer languages, that tell hardware what tasks to perform, how to perform them, and in what sequence. In other words, code is the “brain” behind a computer’s operation.

Historically, this has meant that the companies that create software have good reason to make their code proprietary. This allows them to generate profit by selling licenses to other companies, government agencies, organizations, or to individuals at a contractually-agreed-upon price, for a predetermined period of time, with associated performance, support, and updating services—but without allowing users visibility into the code itself.

“Open-source” software flips the proprietary model on its head. Rather than protecting their code, companies, organizations, and individuals that “open-source” their software purposefully make that code freely available to users—for viewing, using, copying, modifying, and sharing. The volume of open-source code now available is staggering. The largest repository, GitHub, contains hundreds of millions of code libraries contributed by companies, research centers, government agencies, and private citizens.

Contributors on GitHub and elsewhere can make their open-sourcing as restricted or unrestricted as they wish, using licensing agreements to limit access to a specified user-base, opening it up to the public at large, or anything in between. This does not, however, mean that users are able to directly alter the code behind a software program that is actively in use by a company or other organization—this would create serious performance risks and security vulnerabilities.

Open-sourcing might seem counterintuitive for companies that seek to generate revenue through the sale of software. The model does not, after all, protect their code from competitors who might use it in ways that allow them to gain market advantage. But there are substantial benefits to open-sourcing software, too. In license-restricted settings, exposing code can create transparency between the software vendor and the professionals who manage the purchaser’s technology infrastructure. This can allow them to mutually identify and address problems, or to pursue configurations that are especially efficient or that otherwise aid system performance.

Open-sourcing also provides companies with access to a new pool of talent. It allows users with the requisite skills to identify and propose fixes for errors—what the software industry calls “bugs”—to enhance the code’s performance, and to create new features. This allows the company to evolve its product quickly and strategically in response to market signals.

Another advantage of making software open source is that it can encourage adoption by other organizations. In some cases, this leads to new business opportunities for the original developer—for example, through consulting or custom development. In others, it enables reuse or adaptation of the code for new purposes. While this kind of reuse may not generate direct revenue, it can offer reputational benefits.

Broad adoption also increases the number of users who may contribute improvements or build related tools and products. This kind of ecosystem growth can be especially valuable when it accelerates the development of entire industries—such as when Meta open-sourced PyTorch, now widely used in AI research—or when it provides resources that would otherwise be expensive or difficult to build from scratch. The federal government requires all custom-developed source code by or for federal agencies to be reusable by other agencies to prevent duplicative effort and unnecessary spending. It encourages, but does not mandate, open-sourcing.

The technology

The combined government-company team, roughly 25-30 people, began work in March 2020, which makes GAMECHANGER a COVID-19 baby. For the Booz Allen team, the pandemic added a degree of difficulty to what was an already challenging project. As happened in many industries, team members became geographically dispersed, changing work patterns and requiring the program manager to cultivate new team dynamics. The hiring process, which for software developers usually includes an interactive interview that allows the hiring manager to assess a candidate’s problem-solving, programming, and communication skills, also required adjustment.

These people and process changes came on top of the difficulties of working with the government. Deploying software onto a DOD network, in particular, is a notoriously demanding process. It involves time- and labor-intensive documentation, reporting, and scrutiny of the product, all at great length and detail.¹³ The government team’s insistence that Booz Allen use open-source technologies to develop the application, and make its code base open source upon completion, moreover, required the Booz Allen team to implement new internal processes—this was not something the company had done in its work with the DOD before.

The project also required the use of a rapidly evolving AI technique called natural language processing (NLP). Historically, AI systems relied primarily on structured data—inputs that were standardized and numerically encoded. These are the kinds of datasets that are familiar to those who have worked, for example, with numbers entered into Microsoft Excel, or with statistical modeling software like STATA or R. In contrast, modern NLP systems, which have advanced considerably over the past decade, can process human language as input, despite its irregularities, ambiguities, and variable meanings. Like traditional models that use structured data, many NLP models are task-specific; they use a defined dataset and complete prespecified operations. It is important to note that this differs from the large-scale generative models released after GAMECHANGER was developed—such as ChatGPT, Gemini, or Claude—which can generate novel outputs across a wide range of tasks.

GAMECHANGER’s basic functional tasks—to search for and to present language-based documents upon user request—were not a novel application of NLP. It, like other search engines, could largely be built with two programs operating behind a user interface. One of the programs, the crawler, would scan the textual data of the DOD’s policy documents and gather details of what they contain. The other program would use the information gathered by the crawler to build an index: a structured catalog of each document’s language contents that enables them to be ranked based on relevance when a user enters a search term.

By 2018, many commercial companies had developed NLP programs, and the combined team explored the possibility of using one of them for GAMECHANGER. None, however, was assessed—by the combined team or by the companies themselves—to be a viable option. The founder explains this incompatibility as reflecting the fact that DOD policies “are not written in English and so don’t adhere to the usual NLP programming.” The policies, that is, have their own patterns, conventions, and quirks, and these are distinct enough from plain language that they require a custom-built tool. The founder’s vision of a decision aid, moreover, one that could discover and reconcile relationships among policies, required capabilities beyond basic search.

This did not mean, however, that the combined team needed to write the full set of rules and procedures—often referred to as models or algorithms—required to power their desired NLP capability. In fact, avoiding that burden was one of the reasons the JAIC had insisted during contract negotiations that the GAMECHANGER project include the use of open-source software.¹⁴ As explained by a member of the government little team in an interview, “We didn’t need a Ph.D.-level machine learning expert, we needed software from the outside. We just needed to take the best of the free stuff and apply it on government data.” The project’s base functionality upon completion, therefore, would be an original, DOD policy-specific, NLP-powered search interface, developed using publicly available AI tools with documents stored in Advana as its data source.

The build

Software development is the iterative organization, systematization, and coordination of human creativity and skill channeled into computer code. It requires people to work independently, together; to communicate verbally and through documentation; to be disciplined with version control; to participate in planning meetings and post-action reviews; to conduct rigorous testing and quality assurance; to receive user feedback; and then to start the cycle all over again. It is, in other words, very hard to do well.

The most essential element of building effective software is orienting all that human ingenuity and labor toward a clearly defined and well-understood functional objective. Software teams, that is, must retain obsessive focus on what the end product must be able to do. For GAMECHANGER, that singular purpose changed over time. The founder’s vision was to create a very specific, constrained application. It would allow a decisionmaker—in the field, or behind a desk—to enter contextualized information and receive back an assessment of what actions would, or would not, be compliant with applicable authorities, or whether further legal review was necessary to make a final determination.

Such a tool would require great specificity in training the NLP model, together with techniques capable of identifying and interpreting relationships between and among policies.¹⁵ This latter feature would have the ancillary benefit of illuminating correctible discrepancies in the body of DOD policies—surfacing those that are outdated, superseded by other directives, or directly in conflict with each other and in need of remediation. Graphing the connections among policies in this way, in other words, would allow the DOD to prune and curate its guidance. It was this tool that the little team had started to build in the JAIC, and that initially carried over into the work of the combined team.

The combined team started developing GAMECHANGER while also working to ingest unclassified policy documents into Advana. Although GAMECHANGER was being developed using policy documents available in the public domain, it would ultimately need to run on internal, authoritative sources. Advana is an opt-in data lake; the program has no authority to demand that any part of the DOD provide any of its data. This means that members of the combined team needed to persuade the programs and offices responsible for generating and updating DOD guidance to participate. Access to such authoritative data sources is not lightly given within the department, but GAMECHANGER was successful in securing access to 53 of them. While this is a substantial corpus of material, it is nonetheless incomplete; it does not, that is, include all policies relevant to the DOD’s activities and administration.

In 2019, the founder accepted a position of greater responsibility within the JAIC, a transition that left less time and attention available for GAMECHANGER itself. This reduction in the founder’s direct involvement resulted in a shift in the software’s functional objective. Rather than continue to try to build a decision aid, the combined team elected instead to build “a DOD policy Google.”¹⁶

This tool would support search and generate a graph to represent the reference documents associated with a policy, but it would not use AI to conduct reconciliation analysis. Users, that is, could ask GAMECHANGER to find and access a policy, and to get a visualization of other documents referenced in that policy, but that would be the extent of its primary functionality. This functionality would be supported by other tools that collect usage metrics. Advana was equipped to capture data on how many users “clicked into” each of its applications, but it did not capture other items of interest to the GAMECHANGER development team. They wanted more granular data on who was using it, how long they were using it, and what they were using it for, to inform their efforts to improve GAMECHANGER’s performance and refine and add to its capabilities. To do this, the team decided to integrate the open-source software tool Matomo; Matomo was subsequently adopted by and applied throughout Advana.¹⁷

Over time, all the original members of the JAIC little team left GAMECHANGER for other opportunities. Some were burned out, some were looking for a new challenge, and some were pursuing professional advancement. So, too, were changes being made to the institutional structure surrounding and directing GAMECHANGER’s development. There were discussions about whether the JAIC should continue to report through the chief information officer or be directly responsive to the deputy secretary of defense. In 2021, this question became moot when the department released plans to restructure the relationships between and among its primary information technology offices, including the JAIC. At the end of 2021, the decision was made that the JAIC would be absorbed into the Chief Digital and Artificial Intelligence Office, which in summer 2022 assumed responsibility for all JAIC programs, including GAMECHANGER.

Enterprise effects

Making GAMECHANGER a search tool rather than a decision aid changed the development team’s vision of who would use GAMECHANGER and what they would use it for. The founder had envisioned GAMECHANGER users operating in the field or in any position in which decisions about how to apply policy needed to be made quickly and accurately. This would have made GAMECHANGER very broadly applicable—useful to anyone in the DOD who needed to make a determination about whether an action was, or was not, in compliance with policy. These were not the users envisioned for the “DOD policy Google.” The team instead outlined five distinct “personas”—representative types of customers GAMECHANGER would serve. All five personas were civilian and working in positions responsible for policy creation and management, with workflows that included drafting new policies and updating, maintaining, and interpreting existing ones.¹⁸

The decision to make GAMECHANGER’s core features search and document association thus had cross-cutting effects. On the one hand, the combined team’s personas represented a subset of the DOD population and suggested the intent to build a tool that would serve their specific needs. Yet limiting the tool’s functionality to locating policies and identifying connections among them also limited its utility to these same personas. Any tasks that included summarizing, reconciling, or analyzing policies would continue to require desk research. On the other hand, GAMECHANGER’s focus on search made it valuable to populations beyond policy specialists. All DOD employees, after all, occasionally have reason to consult a DOD policy, whether for their work or to navigate their own careers. In other words, GAMECHANGER, as deployed, traded utility for policy specialists for broader adoption.

Users and uses

GAMECHANGER went live in May 2020 when it appeared as an application in Advana, making it accessible to any DOD employee, contractor, or other authorized user with a common access card and access to the Non-classified Internet Protocol Router Network (NIPRNet).¹⁹ Prior to its release, the founder had actively worked to generate awareness and enthusiasm among people with the kinds of workflows that GAMECHANGER would support. This groundwork produced a group of 20 people who used and provided feedback on the application’s earliest iteration, and who thereafter “spread the news” about GAMECHANGER’s existence and capabilities.

GAMECHANGER’s rollout continued primarily through word of mouth; the combined team considered but did not pursue strategic outreach to offices in which staff occupying roles similar to those of the GAMECHANGER personas were likely to work.²⁰ The team supplemented this snowball model, however, with opportunities to engage directly with GAMECHANGER and its developers. The Advana program regularly hosted demonstrations of the data and tools it contained, and GAMECHANGER often participated to highlight updates and new features. The combined team also hosted weekly question-and-answer and tutorial sessions (office hours) and experimented with a GAMECHANGER newsletter. The result was that growth in GAMECHANGER’s user base was not linear but rather occurred in fits and starts that the combined team only infrequently could correlate with an intentional effort to access new populations (Figure 1). Despite this unstructured and informal approach to rollout, GAMECHANGER’s growth between 2022 and 2025 was substantial, moving from roughly 20 unique users to 21,000.

Figure 1

Use of GAMECHANGER itself, however, should not be expected to correlate directly with the addition of new users. Although it is reasonable to anticipate that a larger pool of users will generate a larger number of searches, the timing and frequency of document searches in GAMECHANGER should vary with the timing, frequency, and nature of the work that needs to be completed, and with the occurrence of unanticipated events. Indeed, this is what Booz Allen data reflects and what individual users report. As GAMECHANGER’s user base grew, so did search activity, though it was not linear, and there is no pattern to its peaks and troughs. There is, nonetheless, clear evidence of growth in GAMECHANGER use over time, with search activity increasing steadily between 2022 and 2025.

The data also suggest that the application’s performance has improved. While the number of documents opened per search has remained relatively stable, with an open rate of between roughly 20% and 40%, the number of documents returned per search has declined noticeably—indicating that the software has been refined to deliver more accurate and relevant results. (Figure 2).

Figure 2

Productivity gains and other effects

One of the primary goals for users of AI—whether individuals, organizations, corporations, or governments—is to increase productivity: the amount of work output generated by a given amount of resource input. The relatively recent introduction of generative AI, software that can be used to summarize, synthesize, and combine all kinds of sensory, language, and mathematical data, has begun to show that it can have positive effects on productivity across a broad range of tasks and industries.²¹ GAMECHANGER does not use generative AI, but recent research on the introduction of even non-generative AI-enabled software in the private sector suggests that it is reasonable to expect, if sometimes difficult to measure, productivity gains. In health care, for example, AI-enabled software can now review X-ray and other images very rapidly—much more rapidly than human radiologists—leading to faster results with the same number of machines and the same number of doctors. Similarly, banks are using AI-enabled software to review loan applications, shrinking the period between when a request is received and when a decision is made without any increase in the number of loan officers.

For GAMECHANGER, increasing productivity would mean enabling users to complete policy tasks faster and, therefore, to complete more of them during the same period of time. Indeed, this is what GAMECHANGER’s users report. Prior to its introduction, policy workflows for tasks like policy development or revision, workforce analysis, and compliance review were dominated by knowledge acquired through experience, requests for information from counterparts in other DOD offices, and time-intensive, manual searches through multiple, disconnected systems. GAMECHANGER’s search capability has therefore had a dramatically positive effect on accessibility and timeliness, with users reporting that it has transformed previously hours-long research efforts into minutes-long tasks. One GAMECHANGER user offered this illustration:

“In my policy shop, I have five policy officers, and each focuses on one policy ‘account’—or subject, like human resources. We have 42 agency policies that have to do with HR—promotions, discipline, etc. She has to help research, draft, and maintain those 42 policies. In reality, she works four or five policies at a time. On average, each of those is about 22-25 pages long. In those policies, there’s a section for references. These are the references we based our policy on—public law, financial management regulations, DOD instructions, intelligence community directives, and so forth. … without GAMECHANGER, the officer has to go into the DOD policy portal to search for promotions, she has to search the Office of Personnel Management system, all the intelligence community systems, all manually. So, let’s say she just searches to confirm all those references, and it takes three hours. With GAMECHANGER, she probably can do the same thing in 10 minutes.”²²

A member of the combined team relayed this estimate of the difference GAMECHANGER makes:

“The process before GAMECHANGER for a document review required manually searching for and reading all relevant documents. Because a policy generally requires seven hours of review and is frequently reviewed by 50+ organizations, approximately 1,400 hours are spent reviewing a single policy. There are 12 policies on average coordinated across DOD per month, representing 16,800 hours (700 days) of opportunity to save.”²³

Enhanced productivity is an important potential benefit of AI-enabled software, but there can be other benefits too. Good software can also improve consistency, accuracy, and completeness, attributes of great importance in most, if not all, industries and certainly for work done on and with DOD policy. Here again, GAMECHANGER user reports are positive. All interviewees commented that GAMECHANGER lowers barriers to access, especially for non-policy specialists, and increases their confidence in the quality of their policy products by presenting a more comprehensive set of related and relevant documents than a manual search would have produced. One user explained it this way:

“[GAMECHANGER] is a time and labor saver overall. Very helpful with quality control of the information that goes to the customer. We need to be right. We need to know it’s the best information we can be giving. … If you took GAMECHANGER away today, it would remove a layer of assurance that we’re answering the question properly. … It would remove a method that gets us to the right answer, and it would be a quality decrement.”²⁴

This is not to say that GAMECHANGER users have encountered no errors or omissions, or that they are fully satisfied with its performance. The GAMECHANGER team receives requests for assistance with bugs or other functionality issues, like non-responsiveness, through the Advana Service Desk. Users are also able to ask the team to add new documents—when, for example, a user identifies a policy that has not yet been ingested into GAMECHANGER’s dataset—and for new features, both via the service desk and at bimonthly “office hours” meetings. These can be suggestions for how to enhance the user interface or for the addition of features that supplement or refine search returns. The base product, however—the core functionality of GAMECHANGER—is and will remain a search and association tool; it will not be advanced to include generative AI capabilities.

Implications for AI and innovation at DOD

Much of the GAMECHANGER story bodes well for AI innovation at the DOD. It demonstrates that the department can develop, adopt, and scale AI to address defense-specific workflow problems and to meet business intelligence needs. That GAMECHANGER’s founder and the initial small team were DOD employees shows that the department attracts talented and technically skilled employees. Advana’s centrality to GAMECHANGER’s functionality is evidence that Advana is returning value to the department and that it has the potential to do even more.

The GAMECHANGER contracting relationship with Booz Allen, moreover, confirms that the department can partner creatively with the private sector to mutual benefit. Its use of open-source tools and the decision not to make GAMECHANGER code proprietary are important government and corporate adjustments to modern software development practices. Indeed, these choices have allowed the development team to generalize GAMECHANGER’s code base so that it can be used to meet other DOD needs—most notably enabling search on contract and budget documents—and for use by other government agencies; the Federal Bureau of Investigation and the National Geospatial Agency now have their own versions of a GAMECHANGER clone. Each of these applications requires its own training and tuning, but all of them inherited their core functionality directly from GAMECHANGER.

There are also, however, elements of the GAMECHANGER story that are concerning. The initial government small team members not only left the GAMECHANGER project, but they also ultimately left the DOD. Some were burned out, and some were looking for a new challenge, but all expressed frustration with the extent to which the department’s deep-seated structural and cultural barriers hinder software development. One commented in late 2024 that they still believe that building even simple applications “without involving a private company would be almost impossible. A company has to build and deliver it—there’s no DOD infrastructure to support fully DOD development.”

Others described impediments ranging from the inability to download software onto their DOD-provided computers, to the sometimes-unreasonable limitations imposed by working on secure DOD networks, to the irrelevance of software development to career advancement. Working on GAMECHANGER, one noted, “wasn’t professionally productive” inside the DOD “because nobody was assessing me on software development.” For the department to achieve its goal of accelerating “DoD adoption of data, analytics, and AI from the boardroom to the battlefield,” it will need to address these deficiencies in retaining technically skilled employees—those essential for software development and those needed to provide expert oversight of software acquisition.

Once released, GAMECHANGER’s rollout appears to have been limited in reach. Even those offices for which it would be especially additive—those who work for the policy “personas” the development team identified as GAMECHANGER customers—report having learned about it by word of mouth, and not by direct outreach by the JAIC or the Chief Digital and Artificial Intelligence Office (CDAO). This approach is at odds with CDAO’s mission. GAMECHANGER itself, moreover, is not nearly as powerful a tool as it might have been. The goal of GAMECHANGER’s founder was to use AI-enabled technologies to build a decision aid. The vision was to equip DOD employees with information that would help them to make contextualized, timely, policy-compliant choices about potential courses of action. Such a tool would be tuned to the arcane language of DOD policies and directives, and programmed to find overlap, redundancy, and inconsistency across documents.

This objective changed during development, largely as a result of the founder’s transition into a different role. Short rotations and promotions are not uncommon at the department, and in many cases are beneficial to DOD offices and for individual career advancement. In the context of technology development, however, frequent turnover can hinder momentum—requiring new team members to be brought up to speed—and, in the case of leadership change, can affect the direction of development. For GAMECHANGER, the founder’s promotion meant that the program lost its primary champion. The result was that development was redirected so that what emerged was not a decision tool but instead a search tool with some additional features. GAMECHANGER can find individual policy documents and visually represent policies that the original document includes as a reference. But it does not analyze DOD policies and authorities and rapidly return an assessment of what actions are, and are not, permissible, or that merit additional legal assessment. Neither does it synthesize nor summarize documents—capabilities that have become common in large language model-based generative AI tools.

GAMECHANGER’s functionality was nonetheless most certainly welcomed by its users, who otherwise had to rely on institutional knowledge and time-consuming research to complete their workflows. Its capability, however, has since been eclipsed by publicly available general pre-trained transformers (GPTs) and, increasingly, by other generative AI tools developed for the DOD specifically and for the U.S. government more generally. These tools search DOD policies as well as other materials, and, on that basis alone, they compete with GAMECHANGER for users. Some of them also include chat—the ability to ask questions and receive common language answers—which GAMECHANGER does not. They are not, however, tuned to DOD policy language. This increases the risk that policy products will contain significant omissions and errors, either because relevant materials are missing from the tool’s document repository or because the model is not trained to properly recognize and interpret them. And none of these tools, including GAMECHANGER, will meet the original need identified by GAMECHANGER’s founder.

This is far from an indictment of GAMECHANGER, or of the decision to invest in its development. To the contrary, as an experiment in AI innovation, the GAMECHANGER project must be judged a success. It produced functioning software that has increased workflow efficiency and enhanced employee satisfaction. It broke and replaced outdated processes and challenged detrimental norms. It is proof that the DOD and an important private sector partner can evolve their contracting and development practices. It illuminates problems in talent management, technology infrastructure, and AI dissemination and adoption strategies that the DOD can solve.

It also suggests a path forward as the DOD works to expand its use of modern, AI-powered technologies. The department rightly has and will continue to take advantage of the large and general tools that are emerging rapidly in the private sector. The GAMECHANGER experience suggests that the Pentagon can usefully supplement these with modest but frequent investments in bespoke development projects, done in partnership with technology companies in the defense innovation base, to produce tools that serve DOD-specific workflows exceptionally well. Such a strategy provides access to the open-source functionality that everyone wants without sacrificing the targeted, mission-specific capabilities that defense professionals need.

Authors