Catalyzing Crisis

Though presented individually, in practice the issues described are most likely to lead to catastrophic outcomes when they occur in combination. Taken together, perhaps the most underappreciated feature of emerging catastrophic AI risks from this exploration is the outsized likelihood of AI catastrophes originating from China. There, a combination of the Chinese Communist Party’s efforts to accelerate AI development, its track record of authoritarian crisis mismanagement, and its censorship of information on accidents all make catastrophic risks related to AI more acute.

To address emerging catastrophic risks associated with AI, this report proposes that:

• AI companies, government officials, and journalists should be more precise and deliberate in their use of terms around AI risks, particularly in reference to “catastrophic risks” and “existential risks,” clearly differentiating the two.
• Building on the Biden administration’s 2023 executive order on AI, the departments of Defense, State, Homeland Security, and other relevant government agencies should more holistically explore the risks of AI integration into high-impact domains such as biosecurity, cybersecurity, finance, nuclear command and control, critical infrastructure, and other high-risk industries.
• Policymakers should support enhanced development of testing and evaluation for foundation models’ capabilities.
• The U.S. government should plan for AI-related catastrophes abroad that might impact the United States, and mitigate those risks by bolstering American resilience.
• The United States and allies must proactively establish catastrophe mitigation measures internationally where appropriate, for example by building on their promotion of responsible norms in autonomous weapons and AI in nuclear command.

AI-related catastrophic risks may seem complex and daunting, but they remain manageable. While national security practitioners must appraise these risks soberly, they must also resist the temptation to over-fixate on worst-case scenarios at the expense of pioneering a strategically indispensable, powerful new technology. To this end, efforts to ensure robust national resilience against AI’s catastrophic risks go hand in hand with pursuing the immense benefits of AI for American security and competitiveness.

Introduction

Since ChatGPT was launched in November 2022, artificial intelligence (AI) systems have captured public imagination across the globe. ChatGPT’s record-breaking speed of adoption—logging 100 million users in just two months—gave an unprecedented number of individuals direct, tangible experience with the capabilities of today’s state-of-the-art AI systems.¹ More than any other AI system to date, ChatGPT and subsequent competitor large language models (LLMs) have awakened societies to the promise of AI technologies to revolutionize industries, cultures, and political life. This public recognition follows from a growing awareness in the U.S. government that AI, in the words of the National Security Commission on Artificial Intelligence, “will be the most powerful tool in generations for benefiting humanity,” and an indispensable strategic priority for continued American leadership.²

But alongside the excitement surrounding ChatGPT is growing alarm about myriad risks from emerging AI capabilities. These range from systemic bias and discrimination to labor automation, novel biological and chemical weapons, and even—some experts argue—the possibility of human extinction. The sudden explosion of attention to such diverse concerns has ignited fierce debates about how to characterize and prioritize such risks. Leading AI labs and policymakers alike are beginning to devote considerable attention to catastrophic risks stemming from AI specifically: OpenAI launched a purpose-built Preparedness team to address these risks, just as Anthropic crafted a Responsible Scaling Policy to “require safety, security, and operational standards appropriate to a model’s potential for catastrophic risk.”³ In November 2023, 28 countries signed the Bletchley Declaration, a statement resulting from the United Kingdom's (UK’s) AI Safety Summit, that likewise affirmed AI’s potential to produce “catastrophic” harms.⁴

For national security practitioners, the maelstrom of often-conflicting opinions about the potential harms of AI can obscure emerging catastrophic risks with direct national security implications. Between the attention devoted to the range of harms AI is already causing in bias, discrimination, and systemic impacts on the one hand, and the focus on future-oriented debates about existential risks posed by AI on the other, these emerging catastrophic threats can be easily overlooked. That would be a major mistake: progress in AI could enable or contribute to scenarios that have debilitating effects on the United States, from enhanced bioterrorism to nationwide financial meltdowns to unintended nuclear exchanges. Given the potential magnitude of these events, policymakers urgently need sober analysis to better understand the emerging risks of AI-enabled catastrophes. Better clarity about the large-scale risks of AI need not inhibit the United States’ competitiveness in developing this strategically indispensable technology in the years ahead, as some fear. To the contrary, a more robust understanding of large-scale risks related to AI may help the United States to forge ahead with greater confidence, and to avoid incidents that could hamstring development due to public backlash.

This report aims to help policymakers understand catastrophic AI risks and their relevance to national security in three ways. First, it attempts to further clarify AI’s catastrophic risks and distinguish them from other threats such as existential risks that have featured prominently in public discourse. Second, the report explains why catastrophic risks associated with AI development merit close attention from U.S. national security practitioners in the years ahead. Finally, it presents a framework of AI safety dimensions that contribute to catastrophic risks.

Despite recent public alarm concerning the catastrophic risks of AI, the technology’s integration into high-risk domains is largely still in its nascent forms, especially when speaking of more powerful AI systems built using deep learning techniques that took off around 2011 and act as the foundation for more recent breakthroughs. Indeed, current deep learning–based AI systems do not yet directly alter existing catastrophic risks in any one domain to a significant degree—at least not in any obvious ways. Unanticipated present risks notwithstanding, this reality should elicit reassurance at a time of widespread anxiety about AI risks among Americans, as it gives both the government and industry an opportunity to help guide the technology’s development away from the worst threats.⁵ But this reality cannot encourage complacency: AI may pose very real catastrophic risks to national security in the years ahead, and some perhaps soon. The challenge for national security practitioners at this stage is to continuously monitor and anticipate emerging large-scale risks from AI as the technology rapidly evolves, often in unexpected ways, while the United States continues to ambitiously pursue AI’s transformative potential. To support that effort, this report proposes four key dimensions of AI safety—the technology’s novel capabilities, technical faults, integration into complex systems, and the broader conditions of its development—that will shape the risks of AI catastrophes going forward.