Weighing the existential risks of AI and the proposals to prevent them | The TechTank Podcast

There have been several reported incidents of artificial intelligence (AI) agents going rogue, engaging in malicious activities, and hiding their endeavors from human overseers over the past few months. Some experts have spoken to these risks and discussed the possible implications for our power grids, hospitals, or financial systems if the technology is used by nefarious actors.  

These warnings, including from one former Anthropic employee, have increased attention to the possible “existential threats” some say AI may pose to humanity. Policymakers are now seeking to address these threats and debating what actions should be undertaken by government and business to reassure a nervous public that AI is safe, reliable, and helpful. 

In this episode of the TechTank podcast, co-host Darrell West is joined by fellow Brookings colleagues Tom Wheeler, a visiting fellow in the Center for Technology Innovation, and Bill Galston, the Ezra K. Zilkha Chair in Governance Studies and a columnist for the Wall Street Journal. They discuss various proposals and what they mean for both safety and competition. 

Listen to the episode and subscribe to the TechTank Podcast on Apple, Spotify, or Acast.  

[00:00:00] CO-HOST NICOL TURNER LEE: You’re listening to TechTank, a biweekly podcast from the Brookings Institution exploring the most consequential technology issues of our time. From racial bias in algorithms to the future of work, TechTank takes big ideas and makes them accessible. 

[00:00:26] CO-HOST DARRELL WEST: Thanks for joining our Brookings TechTank podcast. I’m Darrell West, senior fellow in Governance Studies at the Brookings Institution. There have been several recent incidents of AI agents going rogue, engaging in malicious activities, and hiding their endeavors from their human overseers. There furthermore are possibilities for foreign adversaries to use AI in nefarious ways and create what some people have called a digital Pearl Harbor for our power grids, hospitals, or financial systems. These possibilities have led to what a number of people are calling existential threats to humanity. At its most extreme form, experts worry about AI killing people, destroying humanity, or disrupting civilization as we know it. There are questions about how serious to take these threats and what actions should be undertaken by either government and/or businesses to reassure a nervous public. To discuss these important questions, we are pleased to be joined by two distinguished experts. Tom Wheeler is a visiting fellow at the Brookings Institution and the author of several important books on AI and technology innovation. Bill Galston is the Ezra K. Zilkha Chair of Governance Studies at Brookings and a columnist for The Wall Street Journal. He recently wrote a piece questioning the safety of AI systems for that paper. Tom and Bill, welcome to our Brookings TechTank podcast.  

[00:01:58] GUEST BILL GALSTON: Great to be here, Darrell.  

[00:01:59] GUEST TOM WHEELER: Thank you, Darrell.   

[00:02:00] CO-HOST DARRELL WEST: There are many AI disaster scenarios that people are worried about: threats to critical infrastructure, digital Pearl Harbors, rogue agents, and outright annihilation. Tom, let’s start with you. Which of these scenarios do you worry about, and which ones are exaggerated scenarios that shouldn’t worry any of us?  

[00:02:22] GUEST TOM WHEELER: First of all, Darrell, thank you for inviting me. It’s great to be here with you and Bill. I think that the disaster is leaving all the decisions about the AI future in the hands of a few self-selected, self-interested tech bros. The Economist had a great piece a while back in which they asked the question, they said, “Five men control AI. Who controls them?” And I think that the big pending potential disaster really boils down to a democracy question. Our founders created a nation that was able to deal with the experience that they had dealing with autocrats. We went through the Industrial Revolution and developed an industrial policy capable of dealing with industrial autocrats. And now we have a reality where the most powerful technology in the history of the world is governed by policy that is being made by AI autocrats. And so I think we need to come to grips, I hope we as a nation can come to grips with the reality that we need democratic institutions to step up and address the challenge, and there in, and thereby, Darrell, deal with the potential kind of disasters you were listing. 

[00:04:26] I think it is possible to hold three beliefs simultaneously. One is that the safety concerns we’ve all been hearing about regarding AI are legitimate and, should be of concern to everyone. Second is that the leaders of the AI industry have been sincere in their efforts to call out some of those risks, but that sincerity does not abolish self-interest. And then when we talk about what is the democratic role here, we need to understand that self-regulation, self-oversight is also self-interest. And finally, I think the third and most important thing is that we need to recognize that public interest oversight of what’s going on in AI is essential, but that when we develop that policy, as we must, we have to be as innovative as the developers of AI, and not just go back and recreate the old industrial regulatory model, but have a new approach to this new challenge. 

[00:06:04] CO-HOST DARRELL WEST: So Bill, I’d like to get your thoughts on this as well. How worried are you about some of the AI disaster scenarios that have been floating around lately?   

[00:06:17] GUEST BILL GALSTON: I’m, worried, and how worried I am depends on the specific disaster scenario. The scenarios involving somehow the destruction or subordination of the human race to some superintelligence, perhaps I should take that seriously enough to try to put out the fire in my hair, but my hair is not on fire about that. What does disturb me more is the fact that the entire economy, not only of the United States and not just economic production, but just about, everything is now digitally linked in one way or another. And I can see the possibility there of extreme disruptions. And either by, rogue AI agents or foreign entities, including non-state actors, disrupting that for their own malign purposes. 

[00:07:23] But I think the question of how worried I am is less important than the question of how worried the American people are. And they are plenty worried, and it’s a concatenation of worries, they’re, in and around the pell-mell development of AI. First of all, the impact on the economy, and in particular jobs, and in particular jobs for young people who are trying to get a start in this economy. Then of course, there is the nationwide debate about data centers and their impact on everything from electricity prices to the environment and the quality of life. 

[00:08:12] And now the third, the newest and most ominous layer of worry, the one that, that we’re focusing on in this podcast. Given this fact, it’s not just an abstract, need to preserve democratic governance. It’s a very concrete need to respond to the very widespread concerns of the American people. If the U.S. government is not seized of this issue in a serious way, I think that the super majority of people who are worried about all this will feel once again that their governing institutions have failed to address the issues that matter most to them. And I think that is one of the central drivers of the collapse of trust in our governing institutions, the sense that we have problems heaping up, and these institutions are not only not solving these problems, but in many cases aren’t even making a serious effort to do. So I end up agreeing with Tom’s basic proposition that the governing institutions of the country must be seized of this issue and must get on with the discussion sooner rather than later. Otherwise, I think there’s gonna be a massive and justified public backlash.  

[00:09:37] CO-HOST DARRELL WEST: There clearly are serious governance problems here. Congress has not passed any meaningful legislation on AI either to deal with kind of the existential question or even some of the other issues that have been around for many years on privacy, cybersecurity threats, and surveillance, issues. And the speaker just adjourned the House until after the election, so that guarantees there’s not gonna be any legislation basically in the next two months. Soall the public concerns, Bill, that you noted are not–there’s not gonna be a response right away, at least at the, federal level, although there are states that are, taking their own actions. So Tom, we know that one of the issues with generative AI is the fact that it has a built-in learning capabilities. It’s something that its proponents like to call recursive self-improvement. In fact, there was just a New York Times article dealing with this issue. And what that term refers to is the ability to analyze information, to learn new things, and then to act independently on that knowledge. And a lot of people are worried about that particular mechanism, that particular feature of AI. So Tom, I’d like to get your thoughts. How worried should we be about RSI, recursive self-improvement?  

[00:10:59] GUEST TOM WHEELER: Big time worried. I think that this is the breakthrough that started the current frenzy about, concerns about the safety of AI. And the principal example everybody points to, although there have been multiple ones that seem to keep being revealed on a daily basis, but w- back in July when OpenAI was testing some of its agents, they escaped from what was supposed to be an isolated environment and organized together hundreds of them to attack the repository for AI models called Hugging Face, and succeeded in penetrating. And they built this coordinated swarm, if you will, where the process… the, the RSI is, technology that can make a better version of itself without people, and that’s what happened in the Hugging Face example. That they went out and improved themselves on the fly, depending upon what they encountered along the way. And successfully went beyond, redefined their purpose and went beyond what the human instructions were supposed to be. Jack Clark, who was one of the co-founders of Anthropic, wrote last year, that maybe we would see RSI by 2028. I think we have seen it. I think frankly it is why, Dario Amodei and published his pacing proposal that very rapidly, Elon Musk and Sam Altman and Demis Hassabis signed on to, representing other major players. And what they’re trying to say in that proposal is that we need to be able to work collectively to pace the development because where we have come thus far is we have gotten to RSI so quickly we can watch the agents organize collectively but can’t do much about it. 

[00:13:44] And I think this takes us back to where, what, where Bill and I were, what we were talking about, the, whole question of what’s the impact on democracy of recursive intelligence? And let me pick up on the points that Bill was making. there’s, yes, it, there is a technological capability that has come faster than anybody anticipated. But to Bill’s point about the economic challenge of AI RSI is creating its own unique set of economic challenges, which by the way, are if they can be controlled, as close to being controlled by a handful of self-selected individuals, without rules. And back to the political and democratic challenge that Bill was talking about, the question is, who is going to step up and make the rules? I think Bill is right. Democracy requires a response. I think the people are asking for a response, but what we’re getting is a handful of digital barons and AI itself making these decisions rather than democratic institutions, and as Bill points out, the people.  

[00:15:27] CO-HOST DARRELL WEST: No, I agree. Those are all important points. Bill, I’d like to get your thoughts on the ability of AI to learn and to act on that new knowledge.  

[00:15:39] GUEST BILL GALSTON: You’ve put your finger on it, Darrell. That’s what scares me the most because even the people who are in charge of the corporations producing the models that are capable of improving themselves without further human direction have been surprised by the pace of it, and by the content of what’s been produced It’s their business to know, and they don’t. And in a way, that’s the scariest thing of all, because I think a lot of people will be justified in concluding, if they’re surprised, if they really don’t understand how this thing is working, which they obviously don’t, and we then have one more piece of our society that’s out of control. I look at public opinion polls more than I should probably, butyou have a super majority of the American people reporting that they have a sense that things in the country are out of control. This is gonna be exhibit A for that sentiment. And when people get really scared, they do things that they wouldn’t do in more normal circumstances. So I think that the government ought to address itself to the challenge of mitigating the fear that these developments are producing, and in particular, the fear produced by the ability of these machines to improve themselves, not only without a lot in the way of human oversight, but with even less, in the way of human understanding of what they’re doing and why they’re doing it. 

 [00:17:25] It is perfectly true, as you say, Darrell, that Congress is not well set up now to respond quickly. But what about something old-fashioned? A congressional select committee, evenly balanced and with selections that are pointing in the direction of people who have a demonstrated history of being more knowledgeable than their colleagues in the House and the Senate are about this issue, and giving them an accelerated timetable. Instant reaction is not possible, but I don’t think a long neglect of this issue is possible either.  

[00:18:08] CO-HOST DARRELL WEST: You’re right, Bill. I do wanna move from problems to solutions, and, you just, gave us a possible process solution of a select committee that would look into this and make recommendations. And I do think the biggest challenge right now is how we should handle possible, threats, either of the more catastrophic, ones that the public is worried about or the problems related to privacy, cybersecurity, and surveillance that we’ve known about for years. There are a number of ideas that have been proposed, such as industry self-regulation, which is basically trusting the tech companies to protect us from the worst case scenarios, stronger disclosure rules so that when there are problems, at least we know about the rogue agents third-party independent evaluators, independent groups that basically go into companies, oversee what they’re doing, and let us know when there are problems and hopefully address those problems. Outright government regulation, there are different variations on that theme, new agencies or new missions for existing agencies. Or some people have even suggested kill switches that turn off risky applications. So Tom, what is your assessment of the various alternatives that have been proposed to deal with the AI risks? 

[00:19:31] GUEST TOM WHEELER: Yes. I think what I’m saying is that all of them have merits and should not be mutually exclusive. The danger in technology policymaking is that we always tend to define tomorrow in terms of what we knew yesterday. And we also expect that on day one, we’re gonna be able to deliver the perfect solution. This is a learning experience. This is a technology that is moving at exponential pace compared to a society that moves at a linear pace and a Congress that moves at a sclerotic pace. And, and so we need to build into whatever the outcome, whatever the solutions are, and we must have mandatory enforceable solutions the agility to deal with the change that is coming at us so fast. 

[00:20:46] I think that there are basically two cohorts of threats represented by AI. One is humans using AI to do bad things, and we’ve seen that in the increase in fraud, and using it for discrimination, and using it for hacking, and all of these sort of things. I on that point, I think that the former FTC chairman, Lina Khan, was spot on when she said, “There is no AI exemption in existing statutes.” But that means that we need to have aggressive enforcement, and we’re not seeing that. So that’san immediate action that can be taken by the agencies created by Congress. The FTC, the FCC, the FDA, have the authority to move into the marketplace and deal with things that had been established to be bad things that society wants to prevent. And the fact that they are being perpetrated by AI does not change any of that or their power. The second cohort of effects, though, is much more challenging, and that’s AI itself doing bad things, which is what we’ve been talking about thus far today. And we are unprepared to deal with the kind of rogue AI situations that you’ve been talking about. 

[00:22:26] And we are particularly unprepared to deal with the thousands of open weight AI models as compared to the models we think of like OpenAI and Anthropic Claude and, this sort of thing. But the thousands of models that are out there that once they are used, are in the wild and unrecoverable, can’t be called back. And the solution to this AI doing bad things is not to call it a hoax is not to rely on, the stable genius of one man, but to recognize that, hey, we don’t get in an airplane that hasn’t passed a certification test. We don’t get in a car that doesn’t meet safety standards. We don’t move into a home that doesn’t abide by the electrical code And so our challenge is Are we going to let an alien intelligence run wild in two ways? One, have its policies decided by a handful of people, and then two, have it making its own policies? Or are we gonna say that just like we said, electricity is a great thing and we ought to encourage it, but here’s how it has to behave. We’re gonnaestablish those kinds of concepts for AI as well. 

[00:24:14] CO-HOST DARRELL WEST: Yeah. All that makes complete sense. Bill, I’d like to get your thoughts on what you think needs to be done. Of the various remedies that have been suggested, are there particular ones that you think are reasonable and/or effective?  

[00:24:28] GUEST BILL GALSTON: I’d need to be more of a technical expert than I am to answer those questions with the specificity, and the certainty that, the times demand. So let me lay out a few parameters, and one of those parameters is wherever we end up, we need to get started. And what that tells me is that we should get started in a way that doesn’t provide any of the actors in this particular arena incentives to block action altogether. So I’ll give you a few examples. The Department of Defense and the department’s defenders in the Congress are worried about the effect of regulation on our competition with China. That’s a legitimate concern, and I don’t think we should do anything that’s flatly inconsistent with that concern. Details to come. Here’s another design parameter. 

[00:25:38] The industry clearly believes that it has a collective action problem that will render purely voluntary agreements, at best insufficient. despite all of the chatter about regulatory capture and self-interest disguised and, and creating an unlevel playing field for the little guys, I do think that concern which, you know, which the famous Amadei memo and others have expressed very clearly, I think that needs to be taken seriously. And if that involves some sort of, waiver, however temporary, from what might otherwise be the antitrust fears of the major players in this space, so be it. Let’s not let the best be the enemy of the good. Design parameter number three Let’s apply common sense to human motives in these situations. Of course, there’s a substantial measure of self-interest, and so we need mechanisms that are strong enough to override the, policies and responses driven by self-interest that might lean against making progress on this problem. That’swhy I very much favor the idea of embedded third-party evaluators who are not answerable to the tech companies and whose success and failure will be measured in other ways other than the profit and loss of those companies. 

[00:27:26] Fourth design parameter, we need as much information as we can get as fast as we can get it. And it seems to me, based on what I’ve read, that involves much more comprehensive efforts on the part of the tech giants to create internal mechanisms, internal to the operating systems of what they’re designing, that will increase the transparency of what’s going on inside these systems. 

[00:27:54] I am very worried by the reports that I’ve been reading that there’s more that these companies could be doing to monitor, the activities of the systems that they’re creating, but they’re not doing it. If they’re not doing it, whatever the reason, they should be required to do it on a fast track. And fifth and finally, echoing Tom. There’s the situation’s e- evolving so quickly, and there’s so much we don’t know, that we should do nothing that pretends that we’re more certain than we are about what’s going on and what needs to be done, or more knowledgeable than we are. And the regulatory systems that we design on the fly, should be flexible and adaptable. Not our strong suit, particularly at the national level, but we’ve gotta get better at it in a hurry.  

[00:28:48] CO-HOST DARRELL WEST: We clearly need much greater transparency about what is going on. Like right now, we just have a voluntary system. the fact that we’ve learned about some of the recent examples is because companies voluntarily disclosed it. There could be many other examples taking place that we don’t even know about because the companies are not disclosing it. So at a minimum, we definitely need that. So Tom, Bill mentioned the China issue, and of course, industry proponents have claimed the United States should not regulate technology due to our competition with China. They say if America limits AI while China races ahead, the U.S. is going to lose the AI race to China. And I’m just curious, Tom, how do you assess that argument?  

[00:29:30] GUEST TOM WHEELER: I agree with Bill that there are legitimate national security issues involved. I also think that those issues have been used as the shield to justify a lack of action, saying, “Oh, then let’s let these half a dozen companies, American companies, go do whatever they want,” and that’s got us into the current mess. I disagree with Bill on the fact that there needs to be some kind of lessening of competition law, protections. It is quite possible, and in fact, our entire tech economy is built on standards that have been collectively developed and do not violate antitrust laws, and that ought to be possible here. But I think the bigger issue is that we need to consider that this race, quote unquote, with China is really two races, that the United States is running a race to build ever more powerful models And that China is running a race to embed AI applications in their economy using less costly, more efficient models. 

[00:30:56] One of them, the first one, produces runaway rogue AI and monopolistic concentration. The other creates immediate domestic economic advantages and long-term products and services for international economic advantage. But putting it in the context of this discussion, and we have to hope, and as a matter of fact, I wrote something at Brookings today on, on, on this, that the upcoming summit between President Trump and President Xi can begin to address this issue. Of course, as Bill points out, there is a joint fear that among both Beijing and Washington that, the other will gain control of AI and they will be the loser. But there is also a joint reality of what happens when neither can control rogue AI. AI doesn’t really recognize those lines on a map, and if you’re worried about a cybersecurity attack or you’re worried about AI developing biological weapons or other kinds of activities, that’s a worry that both countries have a self-interest in resolving. And so, I would hope that at the upcoming, summit, there’ll be serious discussions about the issue of catastrophic AI risk that recognize that the outcome could be bad for both parties. This does not in any way mean the elimination of the national interest competition, but it does recognize that there was a point in history when, during the Cold War, when adversaries said, we need to have arms control because it’s unsafe for all of us.” 

[00:33:22] CO-HOST DARRELL WEST: Yeah, this should not be seen as a zero-sum game between the United States and China, ’cause I think you’re exactly right. Both countries have a self-interest in avoiding some of the most catastrophic scenarios, that are out there.  

[00:33:34] GUEST TOM WHEELER: But Darrell, the thing we gotta remember when we compare it to the Cold War is that atomic bombs didn’t make their own decisions That’s the difference. That’s why this is a even higher priority than it was historically.  

[00:33:50] CO-HOST DARRELL WEST: Yes. That’s a great point, Tom. Thanks for, bringing that up. Bill, would like to get your, thoughts on the China race and how that affects possible remedies that the United States might consider.   

[00:34:02] GUEST BILL GALSTON: As I said at the outset, I think it’s a legitimate fear, and let me embed that, in a bit of history. Tom at the outset gave us, some history of, regulatory efforts based on, fundamental economic systems, and I’ll just put a proposition on the table that technological competition that’s applicable in the military sphere is one of the main reasons why technological progress occurs in the first place. And you can see in episode after episode throughout human history that whoever gains a decisive technological advantage is able to translate that into a military advantage. So this is a game for high stakes. And, and the country that first got the longbow had an enormous advantage, at least for a while. Similarly, the appl- the application of gunpowder to weaponry. we know what happened in World War I, when the machine gun nullified advantages that, you know, that ground troops had previous- previously had. We’ve seen with our own eyes how transformative drone technology could be. 

[00:35:26] It would be naive to assume that this latest technological surge will not have military applications, and that both of the major powers will not have an interest in capturing those advantages, for themselves in, in what is now a pretty vigorous, competition for military advantage and even military dominance. So granted what Tom said earlier about the different strategies that these, the two great powers, have adopted with regard to the dissemination of AI, I think that there’s still legitimate worry that has to be built into the design of whatever regulatory activity, or structure we may create. Having said that, I’ll just throw out an idea Is it not possible to bifurcate regulation based on the sphere of activity? whether it’s mainly inthe economic sphere or in the military sphere. There are certain things that go on with, nuclear energy in the military sphere, that are subject to forms of regulation that are very different from those in the civilian nuclear sphere. And I wonder, as a structural matter, whether there is some useful analogy here. I’m not sure there is, but I’m not sure there isn’t. If I knew more, perhaps I’d be a little bit more sure.   

[00:37:04] CO-HOST DARRELL WEST: I want to thank each of you, Tom Wheeler and Bill Galston, for sharing your thoughts with us today. each of your considerations of human history and the role of technology innovation and how that should condition our current issues with AI I think are very important. I’m glad that you, raised those points. At Brookings, we write regularly about AI and a range of digital technologies. You can find more information on our Brookings TechTank blog located at brookings.edu. Thank you very much for tuning in. 

[00:37:38] CO-HOST NICOL TURNER LEE: Thank you for listening to TechTank, a series of roundtable discussions and interviews with technology experts and policymakers. For more conversations like this, subscribe to the podcast and sign up to receive the TechTank newsletter for more research and analysis from the Center for Technology Innovation at Brookings.