This report from The Brookings Institution’s Artificial Intelligence and Emerging Technology (AIET) Initiative is part of “AI Governance,” a series that identifies key governance and norm issues related to AI and proposes policy remedies to address the complex challenges associated with emerging technologies.
Artificial intelligence (AI) is a transformative technology that will have a profound impact on manufacturing, robotics, transportation, agriculture, modeling and forecasting, education, cybersecurity, and many other applications. The positive benefits of AI are enormous. For example, AI-based systems can lead to improved safety by reducing the risks of injuries arising from human error. AI-based systems can also make decisions that are more objective, consistent, and reliable than those made by humans. And AI has the power to rapidly and efficiently analyze enormous amounts of data, identifying and making good use of correlations that would elude even the most expert human analyst.
But AI also involves risks. Put simply, AI systems will sometimes make mistakes. A driverless car might fail to avoid an accident that later analysis shows was preventable. An AI-driven algorithm used to evaluate mortgage applications might make decisions that are biased by consideration of impermissible factors, such as race. An AI-enabled robotic surgery tool might take an action during an operation that results in avoidable harm to a patient.
Given the volume of products and services that will incorporate AI, the laws of statistics ensure that—even if AI does the right thing nearly all the time—there will be instances where it fails. While some of those failures may be benign, others could result in harm to persons or property. When that occurs, questions of attribution and remedies will arise. Whose fault is it if an AI algorithm makes a decision that causes harm? How should fault be identified and apportioned? What sort of remedy should be imposed? What measures can be taken to ensure that the same mistake will not be repeated in the future?
“Whose fault is it if an AI algorithm makes a decision that causes harm? How should fault be identified and apportioned? What sort of remedy should be imposed?”
Answering these questions requires examining the intersection of products liability and artificial intelligence. In this policy brief, I provide an overview of key concepts in products liability and their application to AI. I describe the challenges of attribution for AI-induced harms, explain why I believe that products liability frameworks are well-positioned to adapt to address AI questions, and why it is important to promote consistency across states in AI products liability approaches. If implemented with appropriate frameworks, products liability law represents an important mechanism to mitigate possible AI harms.
Products liability in the AI context
Products liability is the area of law that addresses remedies for injuries or property damage arising from product defects, as well as for harms arising from misrepresentations about products. As I explained in a 2014 Brookings Institution paper:
“[Products liability] is a complex and evolving mixture of tort law and contract law. Tort law addresses civil, as opposed to criminal, wrongs (i.e., ‘torts’) that cause injury or harm, and for which the victim can seek redress by filing a lawsuit seeking an award of damages. A common tort, both in products liability and more generally, is negligence. Contract law is implicated by the commercial nature of product marketing and sales, which can create explicit and implicit warranties with respect to the quality of a product. If a product fails to be of sufficient quality, and that failure is the cause of an injury to a purchaser who uses the product in a reasonable manner, the seller could be liable for breach of warranty.”1
Within the broad umbrellas of tort law and contract law, there are multiple specific (and often simultaneous) theories of liability that can be asserted in a products liability claim, including negligence, design defects, manufacturing defects, failure to warn, misrepresentation, and breach of warranty. All of these liability theories can arise in the context of AI. For example, consider the tort of negligence. Manufacturers have an obligation to make products that will be safe when used in reasonably foreseeable ways. If an AI system is used in a foreseeable way and yet becomes a source of harm, a plaintiff could assert that the manufacturer was negligent in not recognizing the possibility of that outcome.
To take a more specific example, consider an AI-enabled system for automatically identifying abnormalities in MRI images, and marketed to medical professionals as a tool for increasing their efficiency at interpreting MRIs. Suppose that the system works well for images with resolution meeting or exceeding a particular precision level, but is unreliable for lower-resolution images. If the maker of this system sells it without providing information about the requisite image resolution, a misdiagnosis on a low-resolution image could be grounds for a products liability claim citing both negligence and failure to warn.
Suppose that, based on the experience of reading thousands of MRI images, the AI system evolves in a manner that makes it better at identifying some abnormalities but significantly worse at identifying others. This could lead to an allegation of a design defect, with the plaintiff arguing that the human designers of the original algorithm could have and should have built the AI system so that it would evolve in ways that would avoid trading off performance enhancements on some abnormalities with performance degradation on others.
Another variant occurs when the manufacturer of the AI-based MRI reading system informs customers that, over time, the algorithm will not only learn from its own accumulated experience in analyzing images, but also from human radiologists who independently review and provide conclusions regarding a randomly selected subset of the images. Suppose further that, due to an oversight by the manufacturer, the feature for including independent radiologist feedback is never incorporated into the AI software; as the algorithm evolves, its accuracy improves less quickly than it otherwise would have. This might lead to a products liability claim asserting that the manufacturer engaged in misrepresentation and the product contains a manufacturing defect. To the extent that a manufacturer makes assurances in the marketing and sale of the MRI reading system that turn out not to true, any resulting harms could give rise to a breach of warranty claim. This would be handled under well-established approaches in accordance with the Uniform Commercial Code, which addresses the explicit and implicit warranties that are created through the sale of goods.
In addition to specific theories of liability based on identifying a particular source (e.g., a manufacturing defect) of a product flaw, another feature of products liability law is strict liability. Under strict liability, manufacturers—including those making AI products—can be held liable for unsafe defects without requiring an inquiry as to whether the defect arose from an identifiable failure, such as a design defect, a manufacturing defect, or manufacturer negligence. Instead, strict liability reflects the view that consumers have a right to expect safe products. When that expectation is not met, a consumer who suffers a resulting harm and brings a strict liability claim would not have the burden of identifying specifically where in the design or manufacturing process the defect was introduced.
An overarching issue that will arise with the growth of AI is an increase in the potential for software-induced harms. In non-AI systems, post-sale software updates have long been a standard approach to fixing product flaws. In many cases, there are no associated harms. For instance, it is common for software security vulnerabilities to be identified and then fixed through updates before being exploited by malicious actors. But there have also been cases where software design issues have reportedly contributed to enormous harms, including two crashes of the Boeing 737 Max, each of which left no survivors. Software was also the reported cause of terrifyingly close call in 1983 when a Soviet early-warning system falsely reported a set of incoming U.S.-launched ICBMs. Fortunately, the officer on duty correctly suspected a software glitch, and no retaliatory strike was ordered.
“As AI becomes ubiquitous across transportation, defense, manufacturing, and many other sectors, the stakes involved in decisions made by AI software will grow.”
While these examples don’t involve AI, they underscore that software decisions can have profound consequences. As AI becomes ubiquitous across transportation, defense, manufacturing, and many other sectors, the stakes involved in decisions made by AI software will grow. That heightens the need to design AI systems in a manner that ensures dangerous flaws are not present when the system is initially placed into the marketplace, and ensures that they’re not inadvertently introduced by the system as the AI algorithms evolve over time.
The Challenge of Attribution for AI Harms
A key characteristic that distinguishes AI is its ability to learn. Stated another way, AI systems don’t simply implement human-designed algorithms. Instead, they create their own algorithms—sometimes by revising algorithms originally designed by humans, and sometimes completely from scratch. This raises complex issues in relation to products liability, which is centered on the issue of attributing responsibility for products that cause harms. After all, if an algorithm designed largely or completely by computers makes a mistake, whose fault is it?
The answer to this question should lie in the recognition that companies need to bear responsibility for the AI products they create, even when those products evolve in ways not specifically desired or foreseeable by their manufacturers. When there are multiple companies that have had a hand in designing an AI system (or in shaping the post-sale algorithm evolution), there can be difficult questions of how to apportion blame when things go wrong. But a defense along the lines of “it’s the algorithm’s fault” won’t be legitimate.
An AI company targeted by a products liability lawsuit will assert multiple defenses. First, it will claim that the AI algorithm isn’t flawed. In addition, it will claim that even if a jury concludes that, despite the company’s belief otherwise, an AI algorithm has evolved in a way that introduced defects,2 that isn’t the responsibility of the company. Instead, the company will argue that since those flaws were introduced after the product had already been placed into the field, blame should be placed elsewhere. The company might try to pin blame for any algorithmic defects on one or more of: the AI itself; the data provided to the system that was used as a basis for the AI-driven algorithm evolution; the human users of the AI system; and/or other companies in the supply chain either downstream or upstream from the company being sued. I consider each of these in turn.
- Blaming AI: As noted above, companies should generally3 not be able to escape liability by blaming AI-driven evolution to algorithms that they originally designed. If companies want to reap the benefits of intelligent algorithms, they also need to be willing to accept the attendant risks. AI enables learning—and therefore, automated post-sale changes to the algorithm aimed at improving its performance. But the anticipation of those future benefits is present at the time of the original sale, and will be reflected in marketing strategies and in product pricing. Thus, even though a company at the time of sale would not know specifically how the AI algorithm might evolve in the future, the fact that it will do so will be portrayed as an asset to prospective customers. If it turns out that, against all expectations, the automatic evolution occurs in a manner that renders the product harmful, the company needs to bear responsibility for that as well.
- Blaming the data: Companies might also assert that there is nothing wrong with the AI system, and that the problem instead lies with bad data that caused it to evolve in harmful ways. Though there is a potential exception (i.e., cases where a malicious user intentionally provides a system with data intended to sabotage its performance),4 generally the “blame the data” argument won’t hold water. Makers of AI systems have a responsibility to anticipate the types of data that might be provided under reasonably foreseeable usage scenarios—and to build in appropriate guardrails so that the resulting algorithm evolution is beneficial or neutral as opposed to detrimental.
- Blaming the users: Depending on the specific facts underlying a particular liability claim, users might have partial or full responsibility for harms arising in association with an AI system. Just as a purchaser of an automobile who drives it at twice the speed limit and then gets in an accident can’t reasonably blame the manufacturer, a user of an AI-based system who applies it in clearly inappropriate ways will bear responsibility for resulting harms. But manufacturers cannot credibly blame users who engage with an AI system in reasonably foreseeable ways and, in doing so, inadvertently cause it to evolve in a manner introducing defects that cause immediate or later harm. The human/AI interface also raises its own set of products liability issues, as building good AI solutions will require an understanding not only of the interactions among the different software components of a system, but also of how humans will interact with those components. Companies that fail to fully anticipate the assumptions and decisions that will shape human/AI interactions risk releasing products that might behave in unintended and potentially harmful ways.
- Blaming the upstream or downstream supply chain: As occurs with non-AI products, products liability in relation to AI will often involve finger pointing at other places in the supply chain. In the AI ecosystem, there will typically be multiple suppliers upstream of a consumer. To start with, there is the company that sold the product directly to the consumer. That company in turn may have purchased a software component of the product from a separate entity. And that entity may have built some portions of the software in-house and licensed other portions from yet another company. Apportioning blame within the supply chain will involve not only technical analysis regarding the sources of various aspects of the AI algorithm, but also the legal agreements among the companies involved, including any associated indemnification agreements.
Products Liability as an Adaptive Area of the Law
As the examples above make clear, the products liability questions that will arise in relation to AI will be highly complex. But that doesn’t mean we need a whole new set of laws to address them. As I wrote in 2014:
“Products liability has been one of the most dynamic fields of law since the middle of the 20th century. In part, this is because the new technologies that emerged over this period have led courts to consider a continuing series of initially novel products liability questions. On the whole, the courts have generally proven quite capable of addressing these questions.”5
In light of this demonstrated history of adaptation to new technologies, products liability law is well positioned to address the questions that will arise in relation to artificial intelligence.
There are several ways in which this adaptation will need to occur. Inquiries into fault in AI-based systems will need to be informed by an understanding that, while the immediate decisions resulting in alleged harms (e.g., a driverless car’s decision to make a turn at a particular instant in time) are made by computers, those decisions can be traced upstream to choices made by companies. That’s where the responsibility will need to be placed when things go wrong. In addition, frameworks for applying risk-utility tests in relation to AI will need to be developed. Risk-utility tests have long been employed in products liability lawsuits to evaluate whether an alleged design defect could have been mitigated “through the use of an alternative solution that would not have impaired the utility of the product or unnecessarily increased its cost.”6 This same test can be applied in relation to AI as well; however, the mechanics of applying it will need to consider not only the human-designed portions of an algorithm, but also the post-sale design decisions and alternatives available to an AI system as it automatically updates its algorithms.
“Inquiries into fault in AI-based systems will need to be informed by an understanding that, while the immediate decisions resulting in alleged harms … are made by computers, those decisions can be traced upstream to choices made by companies. That’s where the responsibility will need to be placed when things go wrong.”
Just as occurs in other areas of products liability, AI liability will generally be handled through state court systems and legislatures.7 Of course, it will take many years to develop a body of case law and statutory law specific to the intersection of AI and products liability, and not all courts will reach the right answer in every case. Over time, though, products liability law will adapt to address the specific questions raised by AI as it has in relation to other emerging technologies.
One way to smooth and accelerate this process as well as to reduce the challenges that can arise due to state-by-state variations is through voluntary frameworks. For example, the American Law Institute (ALI) is a respected organization that produces “scholarly work to clarify, modernize, and otherwise improve the law.” If the ALI or a similar organization were to develop and publish model principles of law and/or legislation specific to AI products liability, this could help to promote greater predictability and uniformity in state-level approaches.
The Brookings Institution is a nonprofit organization devoted to independent research and policy solutions. Its mission is to conduct high-quality, independent research and, based on that research, to provide innovative, practical recommendations for policymakers and the public. The conclusions and recommendations of any Brookings publication are solely those of its author(s), and do not reflect the views of the Institution, its management, or its other scholars.
Microsoft provides support to The Brookings Institution’s Artificial Intelligence and Emerging Technology (AIET) Initiative. The findings, interpretations, and conclusions in this report are not influenced by any donation. Brookings recognizes that the value it provides is in its absolute commitment to quality, independence, and impact. Activities supported by its donors reflect this commitment.
Report Produced by Center for Technology Innovation
- John Villasenor, “Products Liability and Driverless Cars: Issues and Guiding Principles for Legislation” (Washington, D.C.: Brookings Institution, April 24, 2014), 7, https://www.brookings.edu/wp-content/uploads/2016/06/Products_Liability_and_Driverless_Cars.pdf.
- Of course, it is also possible that a plaintiff could also assert that the original algorithm, before any post-sale evolution, was already defective. Those claims would raise many (though not all) of the same issues addressed in this paper.
- One exception to this could occur in the event of algorithmic sabotage by a third party. Though even that exception should not be automatic, as companies have the responsibility to make their algorithms resilient to sabotage attempts that can be reasonably foreseeable.
- Even in the case where a malicious user sabotages an AI system by intentionally introducing data aimed at causing the AI system to behave in a harmful manner, the outcome would still not be the fault of the data. The malicious user would bear responsibility, as potentially would the company for failing to design the system to be more robust against this sort of manipulation.
- Villasenor, “Products Liability and Driverless Cars,” 15.
- Villasenor, 9.
- While products liability is generally handled in state courts, there are some exceptions. See, e.g., the Class Action Fairness Act of 2005, Pub. L. No. 109-2, 119 Stat. 4.