Advancing human control of military AI

The United States and China have not developed a common approach to governing AI in the military domain. This does not mean, however, that they are uninterested in addressing the possibility that some military applications of AI will harm civilians, endanger societies, and undermine global stability.

In late 2024, the presidents of the United States and China that humans, not AI, should have control over the decision to use nuclear weapons. This agreement reflects a shared understanding of the catastrophic consequences of nuclear war, and of the need to minimize the likelihood that it will occur not because leaders have chosen it, but because AI fails or is misused.

As important as that agreement is, it is only the starting point: As militaries integrate agentic AI into their operations and AI models advance rapidly in sophistication and capability, new pathways to catastrophic outcomes are emerging. Presidents Donald Trump and Xi Jinping can lead in governing these dangers by agreeing that only humans, not AI, should make the decision to initiate a cyberattack against each other’s nuclear command, control, and communications systems (NC3) or critical infrastructure.

The dangers of AI-powered cyberattacks 

States rely on NC3 systems to detect threats, make decisions about nuclear use, and transmit secure, reliable launch orders to the forces that would carry them out. Nuclear doctrine does not eliminate these risks. Whether a state relies on no-first-use, launch-on-warning, or other principles, failures in these functions could: cause breakdowns in leaders’ ability to communicate or issue commands; make a state unable to detect or accurately assess threats; cause misperceptions that incentivize a first strike; or result in mistaken or unauthorized nuclear launch. 

An intentional AI-powered cyberattack on another state’s NC3 would therefore initiate a military crisis with possibly catastrophic effects. But an intentional attack is not the only way AI-powered cyber capabilities can cause a military crisis. Such capabilities could also initiate an unintended military crisis in two other ways. First, even if the state’s objective is only to pre-position an offensive cyber capability, an unplanned attack could still result from human error, technical limitations in the AI models, or AI agents that act in ways that humans did not anticipate. 

Second, a bad actor could use the AI-powered cyber capability to execute an unauthorized attack. In both cases, the attacked state would be highly likely to respond by elevating alert levels and readying its nuclear forces. If the disruption to normal NC3 functions were severe enough, leaders would have to make decisions about nuclear employment under conditions of extreme time pressure and deep uncertainty, which might result in nuclear launch and war. 

AI-powered cyberattacks on critical infrastructure—the economic, financial, health, transportation, and safety systems that support civic life—could also result in an extremely serious military crisis. Such an attack could cause the immediate destruction of property, contamination of the local environment, social chaos, and widespread illness, injury, or death. Like attacks on NC3, attacks on critical infrastructure could be initiated purposefully, or as a result of technical error or unauthorized action. A state that suffers an attack on critical infrastructure would be highly likely to retaliate militarily, initiating a war with unpredictable escalation dynamics. 

Presidential leadership is essential

The upcoming U.S.-China AI dialogues are an opportunity for the world’s leading AI countries to meaningfully advance governance of these extremely dangerous uses of AI-powered offensive cyber capabilities. Planned government-to-government dialogues can prepare Trump and Xi to agree that decisions to launch cyberattacks on NC3 and critical infrastructure should be made by humans, not AI. 

Extending the principle applied to nuclear weapons—that attacks with strategic effects should express human intent and must be subject to human accountability—does not require Washington and Beijing to share other principles of cybersecurity or infrastructure protection. It is confined to agreement on the narrow point that both sides have the responsibility to minimize the likelihood that attacks with strategic effects will be caused by chance, error, advanced AI capable of exceeding human-defined constraints, or unauthorized actors. 

Leader-level statements are the necessary precondition for bilateral dialogue on verifiable protocols and safeguards. They are also important public signals that the United States and China recognize the special responsibilities that come with leading in AI and are committed to taking pragmatic steps to reduce the likelihood that their military uses of AI will put nations at existential risk. No other signal carries the same weight in demonstrating that both governments understand what is at stake—not just for their own societies, but for everyone.

AI is penetrating every corner of the military domain at an unprecedented pace. From cyber defense to weapon systems, from intelligence analysis to command and decisionmaking, AI’s presence is ubiquitous. However, the widespread adoption of this technology also brings with it unprecedented risks. In 2024, the leaders of China and the United States reached a consensus on ensuring that nuclear weapons remain under human control at all times. In May of this year, the two countries’ heads of state further proposed building a constructive strategic stability relationship. The governance of military AI should be an integral part of this effort. Yet these political declarations still fall short of preventing AI from spiraling out of control, being misused, or triggering escalations in conflict. China and the United States should explore how to translate their principled consensus into actionable institutional arrangements, while also identifying the practical challenges currently facing them. In future dialogues on military AI, broader issues—including the regulation of lethal autonomous weapons systems—should also be incorporated.

How can the principles be put into practice?

Both China and the United States have repeatedly emphasized the importance of ensuring the principle of “human control,” as reflected in their existing joint statements as well as in their respective policy positions. Melanie Sisson’s article further argues that the authority to launch cyberattacks against nuclear command systems and critical infrastructure must remain firmly in human hands. However, to prevent these declarations from becoming mere formalities, China and the United States must establish practical cooperation mechanisms in three key areas.

First, they should develop a list of red lines for military AI—clearly prohibiting AI from autonomously launching attacks on NC3 systems or autonomously deciding to use nuclear weapons. At the same time, they should also explore a list of critical infrastructure sectors (such as finance, health care, energy, etc.) and clearly define the boundaries.

Second, they should reach a consensus on what constitutes “meaningful human control.” In the 1970s, United States Air Force Colonel John Boyd proposed the renowned OODA loop—in which whoever can “observe, orient, decide, and act” more quickly in a competition or conflict will seize the initiative.  Later, discussions centered on “human-in-the-loop” and “human control” are based on that theoretical foundation. Yet even before Boyd, Chinese Premier Zhou Enlai had already issued instructions demanding “absolute certainty” (万无一失) in China’s nuclear testing efforts. In the view of Chinese experts and practitioners, this instruction continues to hold sway today and embodies the principle of “human control.” Given the two sides’ fundamentally different institutional frameworks and strategic cultures, even when addressing the same issue, they may employ different terminology and adopt divergent approaches. China and the United States need to ensure mutual acceptance of each other’s definitions of “meaningful human control” and related safeguard mechanisms through regular dialogue or by establishing a terminology working group.

Third, they should establish a China-U.S. military hotline for AI. Building on existing China-U.S. military communication mechanisms—such as the defense minister hotline and the Maritime Military Security Consultation Mechanism—adds a dedicated channel for reporting emergency incidents triggered by AI errors or AI cyberattacks. The goal is to reduce the risk of cascading crises caused by AI malfunctions that could mislead humans or even override human judgment. The 2023 balloon incident should be a wake-up call for both China and the United States. Without the safeguard of a hotline mechanism, AI-driven drones and unmanned vessels are highly likely to trigger more accidental escalations in the future.

Greater real-world challenges

Meanwhile, geopolitics and rapid technological iteration—both of which neither side can afford to ignore—are intensifying the security dilemma. Even if China and the United States were to establish a cooperative mechanism on “human control,” they would still face at least three major challenges in practice. The first is the speed dilemma. With the help of AI agents, cyberattacks are unfolding at an unprecedented pace. AI-powered defense systems could automatically trigger countermeasures. Imagine a scenario where an AI detects abnormal traffic targeting the NC3 system, prompting the defensive AI to automatically disconnect the network or launch countermeasures. As a result, the other side might mistakenly interpret this as a cyber intrusion, ultimately pushing both sides’ AI systems into a mode of “automatic confrontation.” AI-enabled cyberattacks can be carried out within milliseconds. In such ultra-high-speed confrontations, the “final human decision” is likely to exceed physiological limits.

Second, there’s the challenge of attribution. It is extremely difficult to trace the origin of cyberattacks. Critical infrastructure can be targeted by a state actor or by nonstate actors such as hacker groups, terrorists, or even AI agents themselves. If critical infrastructure falls victim to an AI-powered cyberattack, how can we determine whether the attack was authorized by the government or whether a nonstate actor has abused AI—or even deliberately framed a state actor to provoke conflict? Without a reliable attribution mechanism, human accountability cannot be effectively established, and retaliatory actions are all too likely to trigger unintended escalations.

Third is the security dilemma. Compared to traditional military technologies, the opacity of AI technologies makes it impossible to accurately assess the actual level of development of an adversary’s AI capabilities, the extent to which these technologies can enhance military operational effectiveness, or even to distinguish whether a particular military AI technology is offensive or defensive in nature. As a systemic undertaking, the effectiveness of AI’s military applications is further influenced by factors such as the software and hardware integration with various subsystems, human-machine collaboration, military institutional mechanisms, and strategic culture—factors that require a relatively lengthy and complex evaluation process. As a result, both sides often imagine their adversaries’ AI deployments based on the worst-case scenario and adopt countermeasures that are equally confrontational. Consequently, threats may be seriously exaggerated, and the spiral of hostility continues to escalate.

These challenges will be difficult to eliminate completely in the near term and can only be managed through continuous strengthening of dialogue and the establishment of safeguards. However, cyberattacks represent merely the tip of the iceberg when it comes to military AI risks between China and the United States; many other critical issues also deserve to be incorporated into bilateral military dialogues. Among these, lethal autonomous weapons systems (LAWS) urgently require effective control. In the Russia-Ukraine theater and the Middle East, the widespread use of autonomous weapons systems has already led to severe civilian casualties and even heightened the risk of nuclear escalation. Once LAWS fall into the hands of nonstate actors, the security risks they pose are immense.

Given the difficulty in achieving progress on LAWS negotiations within the framework of the United Nations Convention on Certain Conventional Weapons, China and the United States should seek a minimum common ground on “prohibiting fully autonomous lethal attacks” and “ensuring human oversight,” and engage in dialogue and cooperation around measures to prevent the proliferation of LAWS and their misuse by nonstate actors. Throughout the history of U.S.-China relations, there have been numerous accidental incidents involving unmanned vehicles or equipment. Most of these systems are deployed in hotspot regions, and once combined with AI and offensive military capabilities, they could very likely trigger unexpected escalations of crises. Both sides should also promptly explore crisis management and response plans for autonomous weapon systems, working together to uphold regional and global strategic stability.

The governance of military AI is not a negotiation that can afford to wait—it’s a race against time. As machines increasingly seize “the right to fire” and even “the right to engage in combat,” China and the United States, as two AI superpowers, bear the historic responsibility of preventing technology from spiraling out of control and safeguarding human civilization. We look forward to seeing the leaders of China and the United States build on the positive momentum of their ongoing AI dialogue during their September meeting, setting an example for global governance of military AI.