1、JUNE 2024Catalyzing CrisisA Primer on Artificial Intelligence,Catastrophes,and National SecurityBill Drexel and Caleb WithersAbout the AuthorsBill Drexel is a fellow for the Technology and National Security Program at the Center for a New American Security(CNAS).His work focuses on Sino-American com

2、petition,artificial intelligence,and technology as an element of American grand strategy.Previously,Drexel worked on humanitarian innovation at the UN(International Organization for Migration)and on Indo-Pacific affairs at the American Enterprise Institute.Always seeking on-the-ground exposure,Drexe

3、l has served as a rescue boat driver during Libyas migration crisis;conducted investigative research in the surveillance state of Xinjiang,China;and supported humanitarian data efforts across wartime Ukraine.He holds a BA from Yale University and masters degrees from Cambridge and Tsinghua universit

4、ies.Caleb Withers is a research assistant for the Technology and National Security Program at CNAS.Before CNAS,he worked as a policy analyst for a variety of New Zealand government departments.He holds an MA in security studies from Georgetown University with a concentration in technology and securi

5、ty,and a bachelors of commerce from Victoria University of Wellington with majors in economics and information systems.AcknowledgmentsThe authors are grateful to Suzanne Spaulding and Andrew Imbrie for their valuable feedback and suggestions on earlier drafts of this report.This report would not hav

6、e been possible without contributions from our CNAS colleagues,including Paul Scharre,Melody Cook,Rin Rothback,Allison Francis,Jake Penders,Tim Fist,Josh Wallin,Michael Depp,and Noah Greene.The report was made possible with the generous support of Open Philanthropy.As a research and policy instituti

7、on committed to the highest standards of organizational,intellectual,and personal integrity,CNAS maintains strict intellectual independence and sole editorial direction and control over its ideas,projects,publications,events,and other research activities.CNAS does not take institutional positions on

8、 policy issues,and the content of CNAS publications reflects the views of their authors alone.In keeping with its mission and values,CNAS does not engage in lobbying activity and complies fully with all applicable federal,state,and local laws.CNAS will not engage in any representational activities o

9、r advocacy on behalf of any entities or interests and,to the extent that the Center accepts funding from non-U.S.sources,its activities will be limited to bona fide scholastic,academic,and research-related activities,consistent with applicable federal law.The Center publicly acknowledges on its webs

10、ite annually all donors who contribute.About the Technology&National Security ProgramThe CNAS Technology and National Security Program explores the policy challenges associated with emerging technologies.A key focus of the program is bringing together the technology and policy communities to better

11、understand these challenges and together develop solutions.About the Artificial Intelligence Safety&Stability ProjectThe CNAS AI Safety&Stability Project is a multiyear,multiprogram effort that addresses the established and emerging risks associated with artificial intelligence.The work is focused o

12、n anticipating and mitigating catastrophic AI failures,improving the U.S.Department of Defenses processes for AI testing and evaluation,understanding and shaping opportunities for compute governance,understanding Chinese decision-making on AI and stability,and understanding Russian decision-making o

13、n AI and stability.TABLE OF CONTENTS01 Executive Summary03 Introduction04 Terms&Concerns07 Clarifying Catastrophe08 The Priority of Addressing AI Catastrophes12 Catastrophic Risks and Dimensions of AI Safety13 New Capabilities15 Technical Safety Challenges21 Integrating AI into Complex Systems21 Con

14、ditions of AI Development24 Further Considerations25 Recommendations27 ConclusionExecutive Summaryhe arrival of ChatGPT in November 2022 initi-ated both great excitement and fear around the world about the potential and risks of artificial intelligence(AI).In response,several AI labs,national govern

15、ments,and international bodies have launched new research and policy efforts to mitigate large-scale AI risks.However,growing efforts to mitigate these risks have also produced a divisive and often confusing debate about how to define,distinguish,and prioritize severe AI hazards.This categorical con

16、fusion could complicate policymakers efforts to discern the unique features and national security implications of the threats AI posesand hinder efforts to address them.Specifically,emerging catastrophic risks with weighty national security impli-cations are often overlooked between the two dominant

17、 discussions about AI concern in public discourse:present-day systemic harms from AI related to bias and discrimination on the one hand,and cantankerous,future-oriented debates about existential risks from AI on the other.This report aims to:Demonstrate the growing importance of mitigating AIs catas

18、trophic risks for national security practitionersClarify what AIs catastrophic risks are(and are not)Introduce the dimensions of AI safety that will most shape catastrophic risks Catastrophic AI risks,like all catastrophic risks,demand attention from the national security community as a critical thr

19、eat to the nations health,security,and economy.In scientifically advanced societies like the United States,powerful technologies can pose outsized risks for catastrophes,especially in cases such as AI,where the technology is novel,fast-moving,and relatively untested.Given the wide range of potential

20、 applications for AI,including in biosecurity,military systems,and other high-risk domains,prudence demands proactive efforts to distinguish,prioritize,and mitigate risks.Indeed,past incidents related to finance,biological and chemical weapons,cybersecurity,and nuclear command and control all hint a

21、t possible AI-related catastrophes in the future,including AI-accelerated biological weapons of mass destruction(WMD)production,financial meltdowns from AI trading,or even accidental weapons exchanges from AI-enabled command and control systems.In addition to helping initiate crises,AI tools can als

22、o erode states abilities to cope with them by degrading their public information ecosystems,potentially making catastrophes more likely and their effects more severe.Perhaps the most confusing aspect of public discourse about AI risks is the inconsistent and sometimes inter-changeable use of the ter

23、ms“catastrophic risks”and“existential risks”the latter often provoking strong disagreements among experts.To disentangle these concepts,it is helpful to consider different crises along a spectrum of magnitude,in which the relative ability of a state to respond to a crisis determines its classifi-cat

24、ion.By this definition,a catastrophic event is one that requires the highest levels of state response,with effects that are initially unmanageable or misman-agedcausing large-scale losses of life or economic vitality.Existential risks are even larger in magnitude,threatening to overwhelm all states

25、ability to respond,resulting in the irreversible collapse of human civili-zation or the extinction of humanity.Both differ from smaller-scale crises,such as emergencies and disasters,which initiate local and regional state crisis manage-ment responses,respectively.While the prospect of existential r

26、isks unsurprisingly provokes pitched dis-agreements and significant media attention,catastrophic risks are of nearer-term relevance,especially to national security professionals.Not only are catastrophic risks less speculative,but the capabilities that could enable AI catastrophes are also closer to

27、 development than those that would be of concern for existential risks.Catastrophic AI risks are also,in many cases,variants on issues that the U.S.government has already identi-fied as high priorities for national security,including possibilities of nuclear escalation,biological attacks,or financia

28、l meltdowns.Despite recent public alarm concerning the cata-strophic risks of powerful“deep learning”based AI tools in particular,the technologys integration into high-risk domains is largely still in its nascent forms,giving the U.S.government and industry the opportunity to help develop the techno

29、logy with risk mitigation in mind.But accurately predicting the full range of the most likely AI catastrophes and their impacts is challenging for several reasons,particularly as emerging risks will depend on the ways in which AI tools are integrated into high-impact domains with the potential to di

30、srupt society.Instead,this report distills prior research across a range of fields into four dimensions of AI safety shaping AIs cata-strophic risks.Within each dimension,the report outlines each issues dynamics and relevance to catastrophic risk.T1TECHNOLOGY&NATIONAL SECURITY|JUNE 2024Catalyzing Cr

31、isis:A Primer on Artificial Intelligence,Catastrophes,and National SecurityThough presented individually,in practice the issues described are most likely to lead to catastrophic outcomes when they occur in combination.Taken together,perhaps the most underappreciated feature of emerging catastrophic

32、AI risks from this exploration is the outsized likelihood of AI catastrophes origi-nating from China.There,a combination of the Chinese Communist Partys efforts to accelerate AI development,its track record of authoritarian crisis mismanagement,and its censorship of information on accidents all make

33、 catastrophic risks related to AI more acute.To address emerging catastrophic risks associated with AI,this report proposes that:AI companies,government officials,and journalists should be more precise and deliberate in their use of terms around AI risks,particularly in reference to“catastrophic ris

34、ks”and“existential risks,”clearly differentiating the two.Building on the Biden administrations 2023 execu-tive order on AI,the departments of Defense,State,Homeland Security,and other relevant government agencies should more holistically explore the risks of AI integration into high-impact domains

35、such as biosecurity,cybersecurity,finance,nuclear command and control,critical infrastructure,and other high-risk industries.Policymakers should support enhanced develop-ment of testing and evaluation for foundation models capabilities.The U.S.government should plan for AI-related catastrophes abroa

36、d that might impact the United States,and mitigate those risks by bolstering American resilience.The United States and allies must proactively establish catastrophe mitigation measures internationally where appropriate,for example by building on their promo-tion of responsible norms in autonomous we

37、apons and AI in nuclear command.AI-related catastrophic risks may seem complex and daunting,but they remain manageable.While national security practitioners must appraise these risks soberly,they must also resist the temptation to over-fixate on worst-case scenarios at the expense of pioneering a st

38、rategically indispensable,powerful new technology.To this end,efforts to ensure robust national resilience against AIs catastrophic risks go hand in hand with pursuing the immense benefits of AI for American security and competitiveness.Safety DimensionQuestionIssues New capabilitiesWhat dangers ari

39、se from new AI-enabled capabilities across different domains?Dangerous capabilities Emergent capabilities Latent capabilitiesTechnical safety challengesIn what ways can technical failures in AI-enabled systems escalate risks?Alignment,specification gaming Loss of control Robustness Calibration Adver

40、sarial attacks Explainability and interpretabilityIntegrating AI into complex systemsHow can the integration of AI into high-risk systems disrupt or derail their operations?Automation bias Operator trust The lumberjack effect Eroded sensitivity to operations Deskilling,enfeeblement Tight coupling Em

41、ergent behavior Release and proliferationConditions of AI developmentHow do the conditions under which AI tools are developed influence their safety?Corporate and geopolitical competitive pressures Deficient safety cultures Systemic underinvestment in technical safety R&D Social resilience Engineeri

42、ng memory life cyclesCNASDC2Introductionince ChatGPT was launched in November 2022,artificial intelligence(AI)systems have captured public imagination across the globe.ChatGPTs record-breaking speed of adoptionlogging 100 million users in just two monthsgave an unprecedented number of individuals di

43、rect,tangible experience with the capabilities of todays state-of-the-art AI systems.1 More than any other AI system to date,ChatGPT and subsequent competitor large language models(LLMs)have awakened societies to the promise of AI technol-ogies to revolutionize industries,cultures,and political life

44、.This public recognition follows from a growing awareness in the U.S.government that AI,in the words of the National Security Commission on Artificial Intelligence,“will be the most powerful tool in gener-ations for benefiting humanity,”and an indispensable strategic priority for continued American

45、leadership.2But alongside the excitement surrounding ChatGPT is growing alarm about myriad risks from emerging AI capabilities.These range from systemic bias and discrimination to labor automation,novel biological and chemical weapons,and evensome experts arguethe possibility of human extinction.The

46、 sudden explo-sion of attention to such diverse concerns has ignited fierce debates about how to characterize and prioritize such risks.Leading AI labs and policymakers alike are beginning to devote considerable attention to cata-strophic risks stemming from AI specifically:OpenAI launched a purpose

47、-built Preparedness team to address these risks,just as Anthropic crafted a Responsible Scaling Policy to“require safety,security,and opera-tional standards appropriate to a models potential for catastrophic risk.”3 In November 2023,28 countries signed the Bletchley Declaration,a statement resulting

48、 from the United Kingdoms(UKs)AI Safety Summit,that likewise affirmed AIs potential to produce“cata-strophic”harms.4For national security practitioners,the maelstrom of often-conflicting opinions about the potential harms of AI can obscure emerging catastrophic risks with direct national security im

49、plications.Between the attention devoted to the range of harms AI is already causing in bias,discrimination,and systemic impacts on the one hand,and the focus on future-oriented debates about existential risks posed by AI on the other,these emerging catastrophic threats can be easily overlooked.That

50、 would be a major mistake:progress in AI could enable or contribute to scenarios that have debili-tating effects on the United States,from enhanced bioterrorism to nationwide financial meltdowns to unintended nuclear exchanges.Given the potential magnitude of these events,policymakers urgently need

51、sober analysis to better understand the emerging risks of AI-enabled catastrophes.Better clarity about the large-scale risks of AI need not inhibit the United States competitiveness in developing this strategically indispensable technology in the years ahead,as some fear.To the contrary,a more robus

52、t understanding of large-scale risks related to AI may help the United States to forge ahead with greater confidence,and to avoid incidents that could hamstring development due to public backlash.This report aims to help policymakers understand cat-astrophic AI risks and their relevance to national

53、security in three ways.First,it attempts to further clarify AIs cat-astrophic risks and distinguish them from other threats such as existential risks that have featured prominently in public discourse.Second,the report explains why catastrophic risks associated with AI development merit close attent

54、ion from U.S.national security practitioners in the years ahead.Finally,it presents a framework of AI safety dimensions that contribute to catastrophic risks.Despite recent public alarm concerning the cata-strophic risks of AI,the technologys integration into high-risk domains is largely still in it

55、s nascent forms,especially when speaking of more powerful AI systems built using deep learning techniques that took off around 2011 and act as the foundation for more recent breakthroughs.Indeed,current deep learningbased AI systems do not yet directly alter existing catastrophic risks in any one do

56、main to a significant degreeat least not in any obvious ways.Unanticipated present risks notwithstanding,this reality should elicit reassurance at a time of widespread anxiety about AI risks among Americans,as it gives both the government and industry an opportunity to help guide the technologys dev

57、el-opment away from the worst threats.5 But this reality cannot encourage complacency:AI may pose very real catastrophic risks to national security in the years ahead,and some perhaps soon.The challenge for national security practitioners at this stage is to continuously monitor and anticipate emerg

58、ing large-scale risks from AI as the technology rapidly evolves,often in unexpected ways,while the United States continues to ambitiously pursue AIs transformative potential.To support that effort,this report proposes four key dimensions of AI safetythe technologys novel capabilities,technical fault

59、s,integration into complex systems,and the broader conditions of its developmentthat will shape the risks of AI catastrophes going forward.S3TECHNOLOGY&NATIONAL SECURITY|JUNE 2024Catalyzing Crisis:A Primer on Artificial Intelligence,Catastrophes,and National SecurityTerms&ConcernshatGPTs release lau

60、nched once-obscure concerns about dangerous,high-impact AI events into the mainstream.Since ChatGPTs arrival,public discourse has seen an unprecedented focus on“existential risks”the fear that AI could wipe out human civilization through a combination of super-human intelligence and misalignment wit

61、h humanitys interests.But the groundswell of public interest in AI-related dangers has also confused the characteriza-tions of these dangers,with experts and policymakers sometimes using terms such as“disaster,”“catastrophe,”and“existential threat”interchangeably,and sometimes to refer to different

62、things.6 The abstract nature of the threats AI poses does not help:unlike nuclear weapons,AI does not explode,and the technologys impactseven if considerableare often indirect.For example,if AI tools are eventually able to help develop a highly lethal pandemic supervirus for nefarious purposes,the r

63、esults could prove much more devastating than any one nuclear strike,even if the crucial role of AI is more subtle.Despite this confusion in terms and concerns,AI-related dangers have firmly established themselves in public consciousness.Fear of extreme dangers from AI motivated thousands of individ

64、uals,including many industry leaders such as Elon Musk and Apple cofounder Steve Wozniak,to issue a statement calling for a minimum six-month pause on building more advanced AI systems in the wake of ChatGPT.The statementwhich suggested a government morato-rium if necessarywas driven by a fear of ru

65、naway AI capabilities posing“profound risks to society and Chumanity.”7 Roughly two months later,a broad coalition of pioneering AI scientists and other notable figures,from OpenAI CEO Sam Altman to Microsoft cofounder Bill Gates,signed a second concise statement expressing similarly grave concerns,

66、asserting that“mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war.”8 Numerous leading publications have published articles exploring calamitous outcomes from advanced AI systems,further escalating fears among t

67、he public and decision-makers alike about devas-tating outcomes from rapidly advancing AI systems.9 A Quinnipiac poll published in May 2023 found that 54 percent of Americans now believe that AI“poses a danger to humanity,”a sentiment echoed in a speech by Vice President Kamala Harris in which she d

68、eclared that AI threats“could endanger the very existence of human-ity.”10 Months later,the UK House of Lords published a report that identified catastrophic risks from AI as an area requiring“immediate attention”from the govern-ment,while simultaneously warning that existential risks from AI were“e

69、xaggerated and must not distract policy-makers from more immediate priorities.”11To disentangle the large-scale threats AI poses,it is useful to distinguish terms such as“disaster,”“catastrophe,”and“extinction-level events.”Within humanitarian and crisis response discourses,these terms have been con

70、tested for years,preventing a common set of definitions.12 Perhaps the clearest way to organize the sometimes-overlapping concepts is by a“continuum of magnitude”in events,as proposed by Clifford E.Oliver,which distinguishes categories according to the scope of their impact and the size and complexi

71、ty of the response required to manage their effects.13EMERGENCYDISASTERCATASTROPHEEXTINCTION LEVELEVENTFigure 1:Continuum of Magnitude14CNASDC4Adapting Olivers and others work,a practical definition of each of these categories comes into focus.15Emergencies are events at a local level that pose a ri

72、sk to the life,well-being,or financial health of one or more individuals.Local officials usually have plans and processes to manage the effects of emergencies,even if their efforts are not always successful.An AI-related emergency might be a self-driving car that accidentally causes a serious road c

73、ollision,requiring immediate medical attention.Disasters are events involving multiple people,large-scale economic damage,or both.Local crisis management resources cannot sufficiently manage disasters,requiring additional support from surrounding localities,regions,or the national government.An AI-r

74、elated disaster could be the malfunction of an automated oil rig system,resulting in the uncontrolled release of millions of gallons of oil into the ocean,similar in extent to the 2010 Deepwater Horizon oil spill.Catastrophes are events of such magnitude in terms of casualties or economic destructio

75、n that they over-whelm the ability of one or more national governments crisis management systems to fully handle their impacts,resulting in unmet critical needs,at least in initial phases.Catastrophes differ from emergencies and disasters not just in terms of size,but also in the nature of their imp

76、acts,as they elicit wide-ranging and interconnected social,political,and economic effects that cannot be managed by any one command and control system.18 Catastrophes thus represent an overwhelming shock to existing social and governmental systems.The Federal Emergency Management Agency(FEMA)offers

77、a similar definition for a“catastrophic incident”:Any natural or man-made incident,including ter-rorism,that results in extraordinary levels of mass casualties,damage,or disruption severely affecting the population,infrastructure,envi-ronment,economy,national morale,and/or government functions.A cat

78、astrophic event could result in sustained national impacts over a pro-longed period of time;almost immediately exceeds resources normally available to local,State,Tribal,and private sector authorities in the impacted area;and significantly interrupts governmental operations and emergency services to

79、 such an extent that national security could be threatened.19 An example of an AI-related catastrophe would be the use of AI to develop a deadly and highly contagious pathogen that wipes out significant swaths of a national population,similar to the effects of the 1918 Spanish Flu.Extinction-level e

80、vents are cataclysmic in scope,threatening to wipe out the human species,such as if a large asteroid were to collide with Earth in such a way that the world became uninhabitable.The risks of such events occurring are often referred to as existential risks in public discourse,but definitions of exist

81、ential risks also include a broader set of scenarios in which an event may not totally wipe out human life,but would“permanently and drastically curtail”the potential of intelligent life or lead to an irrevers-ible collapse of civilization.21 Definitionally,coping with the impacts of such events is

82、beyond the capacity of humanitys collective mechanisms for crisis management.In recent years,existential risks have garnered considerable attention from academics,in part inspired by the work of Oxford phi-losopher Nick Bostrom,who has argued for them to receive far greater attention and resources.2

83、2Some confusion in the terminology around AI risks stems from a sizeable focus on advanced future AI systems as an existential threat to humanity.Several theorists,including Bostrom,have posited that if one or more AI systems could surpass human intelligence,a failure to fully align the systems inte

84、rests with human flourishing could threaten civilization.23 Some proposed scenarios suggest that these risks could play out very quickly as a singular extinc-tion-level event,while others suggest a more gradual process of extinction or crippling disempowerment as humans cede agency and economic vita

85、lity to a superintelligent AI system.24Though existential risks have been a prominent issue in public discourse around AI since ChatGPTs release,the characterization of these risks remains hotly contested by experts.Some view addressing existential risk from AI as a pressing priority due to the very

86、 rapid progress of AI in recent years,the unsolved challenges of reliably con-trolling AI behavior,and the observed ability of AI systems to produce considerable effects in societies already.25 Other experts dismiss the likelihood or intrinsic dangers of developing AI with superhuman intelligence an

87、d express concern that the focus on existential AI risks is distor-tionary or even dangerous in itself.In this view,the specter of existential AI risks is fueled by commercial incentives to hype AI products and sideline more immediate social and legal issues associated with the technology,in combina

88、tion with Luddite and apocalyptic impulses in society that have tended to accompany periods of technological acceleration and social change.26 So great are the disagreements between technologists on the issue that their cantankerous debate was compared to a religious schism in The Economist.27 In an

89、y case,because so many other publications have focused on existential risks from AI,this report will focus instead on catastrophic risks and their relationship to national security.5TECHNOLOGY&NATIONAL SECURITY|JUNE 2024Catalyzing Crisis:A Primer on Artificial Intelligence,Catastrophes,&National Sec

90、urityExtinction-level event:A large asteroid colliding with Earth could cause devastation on initial impact through kinetic energy but may also throw massive amounts of soot and dust into the atmosphere that would block out sunlight for long enough to kill off plant life and collapse food chains.(Ma

91、rc Ward/Stocktrek Images via Getty Images)Emergency:After the 2016 fatal crash of this Tesla Model S near Williston,Florida,investigators at the National Transportation Safety Board found that overreliance on the vehicles autopilot mode,flaws in the autopilot system design,and user error all contrib

92、uted to the crash.16(National Transportation Safety Board via Flickr)Disaster:The Deepwater Horizon Oil spill released four million barrels of oil over 87 days,causing billions in damages.17(U.S.Coast Guard via Flickr)Catastrophe:If scaled to todays population,the Spanish Flu would have killed appro

93、ximately 70150 million individuals globally,or roughly 26 times as many as the COVID-19 pandemic.20(GPA Photo Archive/National Archives via Flickr)CNASDC6Clarifying Catastrophelthough this continuum of risks can help clarify some of the confusion among terms,it comes with important caveats:as the co

94、ntinuum implies,the boundaries between categories are inexact.Events can straddle categories,such as a relatively large disaster or a comparatively limited catastrophe.Crises of various kinds can,and often do,cross borders and classifications as they evolve.Likewise,crisis mismanagement can turn low

95、er-magnitude events into larger and more dangerous ones,for instance,if a poor healthcare system response allows a local outbreak to grow into an epidemic or pandemic.Contributing to the confusion between catastrophic risks and existential risks is the fact that a catastrophe could in principle snow

96、ball into an exis-tential threatwhether of its own accord or through mismanagementeven though the gulf between even a very large catastrophe and a true extinction-level event is much greater than many suppose.28 By focusing on acute public safety risks from discrete events,such as pandemics or indus

97、trial accidents,this continuum does not capture a wide range of more diffuse AI harms in society.For example,risks related to labor automation and job displacement,systemic bias and fairness,mass surveillance,widespread disin-formation,and other diffuse harms canand in some cases already doaffect mi

98、llions of people and have widespread economic,political,and social impacts.Because these harms occur as numerous incidents that are part of a larger,ongoing trend rather than a large,discrete event,they typically are not characterized as“disasters”or“catastrophic events.”This does not make them any

99、less important.In fact,the insidious nature of their harm can sometimes make them more challenging to address relative to discrete large-scale events that have a clear public safety impact.These types of diffuse harms require attention and intervention,both for harms that exist today and for those t

100、hat may materialize in the future,such as rapid-onset labor automation.The types of interventions needed for such issues,however,usually differ considerably from those required to address catastrophic risks,in part because they tend to be more politically fraught and thus necessitate greater social

101、deliberation and coalition building.Given these differences,and the fact that these issues have been covered extensively elsewhere,they are not addressed in this report.29War and CatastropheWars share many of the attributes of catastrophes but are conventionally treated as a distinctif often interre

102、latedissue for a few reasons.For one,whereas catastrophes are primarily managed by crisis management systems,wars are primarily managed by states defense organs,even if there is often overlap in both cases.Additionally,managing war tends to involve a set of strategies and techniques that aim to achi

103、eve specific political ends with their own well-developed corpus of thought,independent of crisis management.30 The techniques for managing catastrophes,by contrast,tend to be directed more toward narrower goals of curbing casualties and economic losses,and returning to a state of relative normalcy.

104、For this reason,and due to the fact that AI disruptions to war have been handled extensively elsewhere,this report does not focus on war itself as a genre of catastrophe.31 Nonetheless,catastrophes can initiate or contribute to the outbreak of wars,as some have argued regarding the drought that prec

105、eded the Syrian Civil War.32 Likewise,some acts of war,especially those that involve civilians and thereby initiate crisis management systems,would also qualify as catastrophes,such as the destruction of a city by air raids,a nuclear strike on a nations homeland,a war-induced famine,or cyberattacks

106、on critical infrastructure.The terrorist attacks of September 11,2001,for example,might simultaneously qualify as a catastrophe and an act of war.ABecause the definitions of emergencies,disasters,and catastrophes are tied to government responses,there is variability between countries as to which eve

107、nts fall into which categories.For example,a hurricane that hits a large area may be classified as a disaster that can be dealt with by regional authorities in the United States,but amount to a catastrophe in a less-developed or smaller nation less equipped to cope with the hurricanes effects,such a

108、s the devastation wrought by Hurricane Mitch in Honduras in 1998.Relatedly,AIs impact on“epistemic security”societies ability to effectively process and act on informationcan also impact the degree to which a state can manage crises.For example,critics on both the right and left have pointed out how

109、 eroded epistemic security adversely affected the governments COVID-19 response.34 Experts fear that current and future AI bots and algorithm-driven information echo chambers could so degrade states crisis response systems that they indi-rectly contribute to catastrophes not by exacerbating the inci

110、ting event itself,but by frustrating the states ability to respond effectively.35 In some instances,states themselves can be the source of catastrophes,even if the definition of“catastrophe”is tied to states ability to respond.For example,Chinas Great Leap Forward was arguably the deadliest catastro

111、phe of the past century,as Mao TECHNOLOGY&NATIONAL SECURITY|JUNE 2024Catalyzing Crisis:A Primer on Artificial Intelligence,Catastrophes,and National Security7In October 1998,Hurricane Mitch wrought devastation across Latin America.In Honduras,about one-third of the population was affected in an even

112、t that Honduran President Carlos Flores Facusse estimated set the country back 50 years in its development.33(Robert Ford via Getty Images)catastrophes that distinguishes them from both emergencies and disas-ters is the lack of a states ability to fully manage the effects of the event.This means tha

113、t beyond causing a devastating loss of life,economic vitality,or both,catastrophes can also threaten the long-term health,security,and stability of the state itself.History is rife with instances of states decline or collapse in the wake of catastrophe,from Athenss plague-induced deterioration durin

114、g the Peloponnesian War to the collapse of Minoan society in the wake of a devastating volcanic erup-tion.37 More recently,the Managua earthquake of 1972,the Bhola cyclone of 1970,and the Ethiopian drought of 197374 all represent instances of natural catastrophes contributing to regime change.38 Tho

115、ugh such catastrophic events are rare,they do occurdespite individuals well-attested tendency to underestimate their likelihood and impacts,a phenomenon known as“normalcy bias.”39 Given catastrophes often-dire conse-quences for states and societies,reducing their likelihood and planning for their ef

116、fects is of utmost importance to policymakers,particularly when technological accelera-tion introduces new risks.In scientifically advanced societies,powerful technol-ogies can often catalyze catastrophes,as in the case of the nuclear meltdown at Chernobyl.The plant meltdown led to the uncontrolled

117、release of 400 times as much radioactive fallout as the U.S.nuclear bomb dropped on Hiroshima,and famously contributed to the polit-ical collapse of the Soviet Union,as Mikhail Gorbachev himself acknowledged.40 In this sense,technological advancement can act as a double-edged sword for developed soc

118、ieties.While naturally occurring eventsin the form of plagues,famines,earthquakes,volcanic eruptions,tsunamis,or hurricaneshave historically been the primary source of large-scale catastrophes for most societies,technological advances have helped blunt many of the worst impacts of natural events.At

119、the same time,however,growing technological capa-bilities have dramatically increased the risks and scope of man-made catastrophes.This trend is evident in war,where technological advancement has enabled the creation of ever-more destructive weaponsfrom sharp stones to crossbows to machine guns and

120、finally nuclear TZedongs state-led drive to transform China from an agricultural society into an industrial powerhouse inadvertently caused between 23 and 55 million deaths from starvation.36 Finally,what catastrophic effects and successful management of those effects look like is,to a degree,sub-je

121、ctive.FEMAs definition of a catastrophic incident,for example,includes extraordinary disruptions to national morale as a sufficient feature of an event to qualify as a catastrophe.While the definition proposed in this report is more narrowly concerned with the magnitude of casu-alties and/or economi

122、c destruction as core indicators of a catastrophe,it is worth considering that alternative definitions may more highly prioritize other features that often go hand in hand with large-scale losses of lives or economic vitality.The Priority of Addressing AI Catastropheshe risks of AI-related emergenci

123、es,disasters,catastrophes,and extinction-level events are all worthy of attention as AI technologies mature,but catastrophic risks are particularly relevant to national security policymakers for several reasons.For one,given that the effects of catastrophes overwhelm the response capacities of all s

124、ub-national authorities,national security practitioners bear much of the primary responsi-bility of addressing such events.Moreover,one feature of CNASDC8weapons with the capacity to kill billions.Outside of advances in weaponry,too,the trend holds.Humanitys growing ability to wield increasingly pow

125、erful technologies creates the potential for ever-greater catastrophes from inadvertent civilian applications of technologyfrom possible leakages of dangerous patho-gens to nuclear reactor meltdowns to human-caused ecological collapse.41 with effects of extraordinary magnitude,such as nuclear reacto

126、r meltdowns and sophisticated bioweapons.This dynamic is most acute when technologies are in their infancybefore the risks are fully understood and cor-rective measures are established over time through trial and error.45With the partial exception of AI-powered autonomous weapons,the destructive pot

127、ential of AI may not be as readily apparent as that of some other technologies.Even if AI tools do not explode like nuclear bombs,AI systems more subtle and complex hazards may be no less profound.Like electricity,AI is a general-purpose technologyable to be used in a vast array of applica-tionsand

128、is being rapidly integrated into complex,delicate systems from healthcare to global logistics,as well as unlocking scientific breakthroughs in multiple fields.Since 2019,private investment in AI development has exceeded$100 billion per year and may well rise further in the near term,accelerating AI

129、progress and deployment.46 The increasing capabilities of AI systems,whose inner workings are often inscrutable to human oversight and sometimes superior to human abilities,means that sophisticated AI tools in combination with other systems and technologies could significantly alter the risk profile

130、 of hazardous domains.The speed of AI deployment,the diversity of potential applications,and the quickly growing capabilities of AI models all lend themselves to heightened catastrophic risks in a variety of fields.Some incidents have already demonstrated a proof of concept for possible catastrophic

131、 risks in which AI plays a role.AI tools have demonstrated the ability to aid in the design and manufacture of chemical weapons,sug-gesting a potential future in which nonstate actors can more easily develop and launch chemicaland perhaps eventually biologicalattacks.47 In 2010,algorithmic trading l

132、aid the foundation for a“flash crash,”causing a trillion dollars to be temporarily wiped out of the stock market.48 With the chair of the Securities and Exchange Commission(SEC)warning that AI“will be the center of.future financial crises,”far more debilitating crashes may well be on the way.49 Auto

133、mated military systems used in nuclear command and control have also suffered failures and false alarmsincidents that some fear could portend AI-induced nuclear catastrophe scenarios in the near future.50 As government leaders grapple with the dangers AI poses,it is important to better understand po

134、tential risks of AI catastrophes.51Given these precedents,corollary fears of AI-enabled bioterrorism,runaway cyberattacks,financial melt-downs,and nuclear misfires naturally represent the A helicopter view of the destruction at the nuclear plant in Chernobyl,Ukraine,just a few days after the meltdow

135、n in April 1986.(Vladimir Repik/AFP via Getty Images)If war is any indicator,the overall impact of techno-logical advancement ostensibly heightens the relative priority of mitigating catastrophic risks relative to disasters and emergencies.Whereas the exponential progress in weapons destructive capa

136、city has been partially offset by advances in medicine and defense technologies in terms of fatalities in conventional conflicts,states still have limited options to manage the risks of more extreme events,such as nuclear strikes.44 A similar dynamic could be at play in civilian uses of powerful tec

137、hnologies.Technology has provided tools to more effectively manage emergencies and disasters of a more limited nature.But technology has simultane-ously escalated the dangers of large-scale catastrophes by unleashing extremely destructive forces upon society 9TECHNOLOGY&NATIONAL SECURITY|JUNE 2024Ca

138、talyzing Crisis:A Primer on Artificial Intelligence,Catastrophes,and National Security200M50M10M2M500K100K20K5K1K2005010Antiquity 1100 1200 1300 1400 1500 1600 1700 1800 1900Theoretical Killing Capacity Per HourTheoretical Killing Capacity Per HourYear(sometimes approximate)ENGLISH LONGBOWENGLISH LO

139、NGBOWENGLISH LONGBOWCROSSBOWCROSSBOWCROSSBOWHAND-TO-HANDHAND-TO-HANDHAND-TO-HANDJAVELINJAVELINJAVELINORDINARY BOWORDINARY BOWORDINARY BOWARQUEBUSARQUEBUS17TH CENTURYMUSKET17TH CENTURYMUSKET17TH CENTURYMUSKET18TH CENTURYFLINTLOCK18TH CENTURYFLINTLOCK18TH CENTURYFLINTLOCKEARLY 19THCENTURYRIFLEEARLY 19

140、THCENTURYRIFLEEARLY 19THCENTURYRIFLEMID 19THCENTURYRIFLEMID 19THCENTURYRIFLEMID 19THCENTURYRIFLELATE 19THCENTURYRIFLELATE 19THCENTURYRIFLELATE 19THCENTURYRIFLESPRINGFIELDMODEL 1903RIFLESPRINGFIELDMODEL 1903RIFLESPRINGFIELDMODEL 1903RIFLEGRIBEAUVAL18TH CENTURY12-POUNDER CANNONGRIBEAUVAL18TH CENTURY12

141、-POUNDER CANNONGRIBEAUVAL18TH CENTURY12-POUNDER CANNONWORLD WAR I MACHINE GUNWORLD WAR I MACHINE GUNWORLD WAR I MACHINE GUNWORLD WAR II MACHINE GUNWORLD WAR II MACHINE GUNWORLD WAR II MACHINE GUNWORLD WAR I FIGHTER-BOMBERWORLD WAR I FIGHTER-BOMBERWORLD WAR I FIGHTER-BOMBERWORLD WAR ITANKWORLD WAR IT

142、ANKWORLD WAR ITANKFRENCH 75MMGUNFRENCH 75MMGUNFRENCH 75MMGUN105MM HOWITZER105MM HOWITZER105MM HOWITZER155MM GPF155MM GPF155MM GPFWORLD WAR IIMEDIUM TANKWORLD WAR IIMEDIUM TANKWORLD WAR IIMEDIUM TANK155MM LONG TOM155MM LONG TOM155MM LONG TOMWORLD WAR II FIGHTER-BOMBER(P-47)WORLD WAR II FIGHTER-BOMBER

143、(P-47)WORLD WAR II FIGHTER-BOMBER(P-47)ONE-MEGATONONE-MEGATON NUCLEAR AIRBURST NUCLEAR AIRBURST20-KILOTON NUCLEAR AIRBURST20-KILOTON NUCLEAR AIRBURST20-KILOTON NUCLEAR AIRBURSTV-2 BALLISTICMISSILEV-2 BALLISTICMISSILE17TH CENTURY 12-POUNDER CANNON17TH CENTURY 12-POUNDER CANNON17TH CENTURY 12-POUNDER

144、CANNON16TH CENTURY 12-POUNDER CANNON16TH CENTURY 12-POUNDER CANNON16TH CENTURY 12-POUNDER CANNONIndividual missile weaponsIndividual missile weaponsIndividual hand-to-hand weaponsIndividual hand-to-hand weaponsRifled small armsRifled small armsSmoothbore small armsSmoothbore small armsMachine GunsMa

145、chine GunsSmoothbore artillerySmoothbore artilleryRifled artilleryRifled artilleryTanksTanksTanksCombat aircraftCombat aircraftCombat aircraftGuided missilesGuided missilesGuided missilesNuclear weaponsNuclear weaponsARQUEBUSONE-MEGATON NUCLEAR AIRBURSTFigure 2:Growth in Weapon Lethality over Time A

146、s technology has improved,the destructive capacities of weapons have increased over time.Though theoretical,Trevor Dupuys attempt to quantify the lethality of weapons based on range,rate of fire,accuracy,reliability,radius of damage,and other factors gives some indication of the growth of destructiv

147、e capacity in weapons driven by technological advancement.42 Alexander Kotts efforts to explore the performance power of direct-fire weapon systems over centuries suggests a similar story of the exponential growth of destructive power.43 Chart by Trevor N.Dupuy,The Evolution of Weapons And Warfare.A

148、dapted by Bill Drexel and Caleb Withers;Design:Melody Cook/CNAS.CNASDC10CNASDC10CNASDC10scenarios that garner the most attention for near-to medium-term catastrophic risks.There are very good reasons to focus attention on each case.But the clear recognition of these specific fears may also help to c

149、urb their likelihood.By contrast,responding to more unex-pected developments that attract less attention over time may ultimately prove more challenging,high-lighting the importance of building awareness of and resilience to a more holistic set of AI safety dynamics.Indeed,these and other risks of c

150、atastrophe in high-impact domains,such as biosecurity,cyber-security,finance,autonomous weapons,high-risk industries,critical infrastructure,and nuclear command and control are far from static regardless of AI developments,and depend on scientific,techno-logical,political,and social changes in each

151、domain.Assessing AI-related risks in any one domain thus involves the interplay between two moving targets:the changing risks of the domain itself and rapidly developing AI capabilities.Additionally,the relation-ship between scientific progress in AI and high-risk domains such as biotechnology or cy

152、bersecurity often exhibits a synergistic effect,as new capabilities in one field can unlock new capabilities in the other.This amplification effect is referred to as“technological convergence”and adds another layer of complexity to characterizing risk in these domains.52 Due to this complexity,unfor

153、eseen developments in any domain could alter pathways to catastrophe in unpredictable ways.Despite this uncertainty,the considerable work that has been done on how technological progress interacts with safety risks,and the sub-discipline of AI safety in particular,can help illuminate how AI devel-op

154、ment can impact catastrophic risks with national security implications.A final reason why AI catastrophes are worthy of attention is that in addition to potentially exacer-bating the chances and severity of catastrophes in a variety of domains,AI could also make it even more difficult for states to

155、manage their effects.As men-tioned previously,some experts fear that AI tools such as deepfakes,LLMs,and more sophisticated recommendation algorithms could considerably degrade societies information environments,in turn degrading their crisis response capabilities.53 In this view,a combination of mo

156、re convincing,personalized,and abundant mis-and disinformation created from AI tools and greater media polarization from siloed,AI-fueled media subcultures could make citizens more susceptible to false narratives.Such an environment would inhibit the ability of states to make and execute decisions i

157、n times of crisis,and would erode public trust generally over time.Already,LLMs have shown poten-tial in lowering the cost and enhancing the quality and scale of disinformation operations,and deepfakes are being deployed in high-profile cases to influence conse-quential political processes.54 While

158、much of the work on these issues has focused on risks to open,democratic media ecosystems,AI tools could have parallel effects in autocratic systems,albeit through different means.Rather than sowing distrust and confusion,autocrats use of AI to bolster propaganda and censorship could exacerbate the

159、challenges of information distortion that plague autocratic regimes,in which critical information fails to reach autocratic leaders,who in turn make poor decisions that can exacerbate or initiate crises as they begin to believe their own propaganda.55 The Great Leap Forwardthe largest catastrophe of

160、 the past century by number of casualtieswas in large part fueled by such information distortion,suggesting that despite the outsized focus on open societies,AIs impact on the information ecosystems of closed societies may be more severe in terms of catastrophes.56 AIs impact on information environm

161、ents is a risk factor that differs in kind from the AI safety dynamics that are the primary subject of this report insofar as it acts as an overarching concern that could affect AI-related crises in any domain,as well as crises that emerge independently of AI tools.Given that a states relative abili

162、ty to respond to a disruptive event ultimately determines the extent of the events impactsand that catastrophes are often the result of state mismanagement of smaller-scale disastersthe influence of AI-powered media degradation on cata-strophic risks could be considerable.The increasing capabilities

163、 of AI systems,whose inner workings are often inscrutable to human oversight and sometimes superior to human abilities,means that sophisticated AI tools in combination with other systems and technologies could significantly alter the risk profile of hazardous domains.11TECHNOLOGY&NATIONAL SECURITY|J

164、UNE 2024Catalyzing Crisis:A Primer on Artificial Intelligence,Catastrophes,and National SecurityCatastrophic Risks and Dimensions of AI Safetyith the exception of a handful of specific proposed scenarios discussed in the following sections,for the most part AI catastrophic risks of relevance to nati

165、onal security are still taking shape.Because advanced AI applications in high-im-pact domains are mostly in their infancy,much of the concern about AI catastrophes today is prospectivea well-informed intuition that the vast power of AI likely could result in tremendous hazards once applied to conseq

166、uential arenas,even if the largest risks have yet to materialize.Nonetheless,given the rapid pace of AI advancement and considerable scope for the impacts of AI in national security,considering how AI safety could impact catastrophic risks as the technology develops is indispensableoffering the oppo

167、rtunity to guide the technologys development toward safety and stability to the extent possible,rather than retroactively addressing severe risks after they have emerged.A clearer awareness of the underlying dynamics driving catastrophic risks related to AI can help build resilience and reduce the c

168、hances of experiencing a major AI catastrophe.In the service of helping to shape preparation for catastrophic risks of AI even as the technology develops,this report proposes four broad dimensions of AI safety as they relate to catastrophic risk.These dimensions distill the insights of a range of bo

169、th AI-specific and broader literature on safety and risk,aiming to be flexible enough to apply across a wide range of domains.57 To further explore these categories,subissues are identified in each area.Though presented independently,in practice these issues often overlap.After clarifying each subis

170、sue,its relevance to cata-strophic risks in national security is examined.Although existing incidents and precedents are cited where possible in these explorations,many of the scenarios proposed are largely hypothetical,and some may not be relevant for many years to come,if ever.Additionally,these t

171、hemes are narrowly focused on understanding the set of issues that contribute to AIs catastrophic risks,and do not include solutions.In all cases,researchers and engineers are working to address these dynamics,but to recount that work is beyond the scope of this report.The issues considered here aim

172、 not to be exhaustive,but to provide a foundation with key examples and refer-ences to broader safety literature as a means to more holistically assess how AI can shape catastrophic risks as the technology evolves and is increasingly built into consequential systems.It is important to note that this

173、 exploration is not intended to provide a full risk management assessment for any particular scenario,which is traditionally a three-step process:1.Assessing risk as a factor of likelihood(including both threats and vulnerability)and consequence.2.Considering mitigations for threats,vulnerabilities,

174、and consequences.3.Prioritizing mitigations that most reduce overall risk.Given how broad and fast-moving AI applications are,and the fact that the rollout of advanced AI capabilities across domains is largely still in its infancy,accurately assessing the full range of the most likely AI cata-stroph

175、ic threats,vulnerabilities,and consequences is simply not possible.Threats and vulnerabilities will vary widely between domains,and will evolve over time depending considerably on how deeply AI tools are integrated into high-impact systems that have the potential to disrupt society.58 Systems associ

176、ated with biological security,cybersecurity,financial security,militaries,high-risk industries,and critical infrastruc-ture are the most obvious candidates,but there may well be others.Given the immense promise that AI systems hold,there is good reason to believe that they may eventually become high

177、ly integrated into any or all WDimensionQuestion New capabilitiesWhat dangers arise from new AI-enabled capabilities across different domains?Technical safety challengesIn what ways can technical failures in AI-enabled systems escalate risks?Integrating AI into complex systemsHow can the integration

178、 of AI into high-risk systems disrupt or derail their operations?Conditions of AI developmentHow do the conditions under which AI tools are developed influence their safety?CNASDC12of these domains.But the timing and conditions under which such integration occurs is an open question and will vary.In

179、 most cases,trying to assign specific likelihoods to not-yet-developed systems would be premature.What may prospectively seem like the most obvious high-risk scenarios in a sector are often also the most likely to be addressed early,meaning that the very act of clearly identifying a specific pathway

180、 to catastrophe may reduce its likelihood of occurring.But even pre-dicting“likely”scenarios early is a challenge:reality so often proves stranger than fiction,contingent on unpredictable forces and extraordinary courses of events.59 Rather than providing risk management assess-ments themselves,this

181、 report aims to help lay a foundation for future risk management assessments,which will require continuous updating and more granular attention to specific scenarios based on a range of variables,including:Risk types:misuse(e.g.,AI-enhanced bioweapons),accidents(laboratory leaks),or structural issue

182、s(widespread poor biosafety controls due to insuffi-cient safety research)60 Specific domains(cybersecurity,biosecurity,finance,nuclear stability,autonomous weapons,high-risk industries)Actors(lone wolves,terrorist organizations,states,corporations)Incentives(terror,prestige,profits,regulatory envir

183、onments)Types of AI models(“narrow”models vs.general-use or“foundation”models)To examine more specifically how these AI safety dimensions manifest in a particular domain of interest,this report will be paired with a follow-on report,AI and the Evolution of Biological National Security Risks.A final

184、word of caution is in order before delving into the many dynamics that could contribute to AI catastrophes.Restricting this primer only to the possible dangers stemming from AI runs the risk of fostering an excessive fixation on what could go wrong,rather than an affirmative vision of what could go

185、right.Readers should avoid this distortion.The opportunity costs of failing to proactively pursue AI development,while impossible to measure,could be severe.As societies become more complex,leveraging AI to help manage their complexity will likely be an overall boon to reducing catastrophic risksnot

186、 to mention the tremendous potential of AI to enhance Americas economy and national security.Relatedly,falling behind China,an adversary with the stated goal of supplanting the United States leading position in AI,also represents a severe risk.61 As further described below,not only would Chinese pre

187、eminence in AI grant Beijing strategic economic and military advan-tages over the United States and help bolster autocratic rule around the world,it would also greatly exacerbate the likelihood of AI catastrophes generally.62 For these reasons,it is imperative that the United States continue to bold

188、ly pioneer the development of AI technologies.Highlighting the dynamics of AI catastrophic risks is in the service of that goalnot an admonition against ambitiously building powerful,effective AI tools.New CapabilitiesNew capabilities from AI tools can have dangerous impacts across a range of domain

189、s,either directly from AI systems themselves or from AI-related breakthroughs in adjacent scientific or technolog-ical domains.These dangers are most prominent in relation to cyber,epistemic,biological,and chemical security,where sudden new capabilities could have dramatic effects,and in some cases

190、disrupt existing deterrents and technical or financial barriers that serve to mitigate the risks of catastrophe.DANGEROUS CAPABILITIESA range of AI tools exhibit hazardous capabilities of relevance to several high-risk domains that experts anticipate will become only more powerful as the technology

191、progresses.Models can produce mis-and disinformation at scale and with increasing quality,posing a threat to societies information ecosystems.63 In biological and chemical applications,AI systems have shown potential in helping to develop weaponiz-able chemicals or pathogens(although not necessarily

192、 aiding actors any more than existing tools,and in dif-ferent ways depending on the preexisting expertise of users).64 Cybersecurity professionals also see growing use of generative AI in phishing attacks and anticipate more sophisticated AI capabilities on the horizon.65 Additionally,seemingly beni

193、gn AI-enabled capabilities could have hazardous implications,such as advance-ments in material science,jet propulsion,or other fields that could be repurposed for weapons use.13TECHNOLOGY&NATIONAL SECURITY|JUNE 2024Catalyzing Crisis:A Primer on Artificial Intelligence,Catastrophes,and National Secur

194、ityImplications for Catastrophic RiskExperts have warned that dangerous capabilities from emerging AI tools could raise the likelihood,severity,or both,of catastrophic attacks in both biosecurity and cybersecurity.In the former,general-use foundation models could lower the barriers to entry for bad

195、actors seeking to build or procure high-impact bioweapons,while AI-powered“biological design tools”may even-tually help craft more strategic or deadly biological agents.66 Cybersecurity experts have likewise warned that AI tools could make cyberattack capabilities more broadly accessible,and enhance

196、 the quality and sophis-tication of advanced cyberattacks,potentially targeting critical infrastructure with catastrophic effects.67 Cyberattacks could also target emergency response communications systems,further exacerbating the impacts of crisis events.But sudden,new capabilities related to AI ad

197、vance-ments could also exacerbate the risks of catastrophe in less direct ways.As previously mentioned,the use of LLMs and other tools may degrade a states ability to cope with disasters or catastrophes by facilitating higher-volume and better-quality misinformation and disinformation at scale.68 Ad

198、ditionally,AI technologies tendency to usher in sudden breakthroughs in a wide variety of scientific subfields could lead to the sudden introduction of strategically disruptive new capabilities that escalate the chances of miscalculations in high-stakes domains,including conventional or even nuclear

199、 deterrence.69 In such cases,where stability is predicated on a degree of confidence about capabilities on both sides,sudden new capabilities can upend the delicate equilibrium of actions and reactions that is fundamental to stability.For example,nuclear stability could be greatly impacted if one co

200、untry unexpectedly developed an AI tool able to crack encryption systems protecting nuclear command and control systems abroad,or if AI-enabled breakthroughs in nuclear delivery systems offered one country a sudden,significant advantage over adver-saries.Although these are provocative examples,more

201、subtle gradations of this dynamic are possible across a range of domains.EMERGENT CAPABILITIESThe capabilities of foundation AI models have steadily increased over time alongside the exponential growth of compute used to train them.70 However,specific capa-bilities can emerge suddenly,improving shar

202、ply from minimal to strong competence as models are scaled.These capabilities can emerge at seemingly unpredict-able points and without specific encouragement from model developersalthough researchers have con-tested the degree to which such capability jumps are truly surprising,or simply a mirage o

203、riginating from the methods used to measure capabilities.71 In practice,this means that the specific capabilities of newly devel-oped models often cannot be fully anticipated before training:disruptive or destructive capabilities may fall into the hands of developers who were not seeking or preparin

204、g for them,posing challenges for the manage-ment of strategic and potentially risky applications.72 Implications for Catastrophic RiskUncertainty about the timing and nature of dangerous AI capabilities as they emergeoften as unintended byproducts of the pursuit of other capabilitiesfurther complica

205、tes states abilities to mitigate catastrophic risks from emerging AI systems.The warning signs of emerging risks that accompany more incremental,pre-dictable technological development may be less regular or less pronounced in the case of AI,making it difficult to develop safeguards ahead of systems

206、proliferation.LATENT CAPABILITIESAI models capabilities may not always be detected by their creators,such that dangerous capabilities may only become known when stumbled upon by othersincluding,perhaps,malicious actors.For example,foundation models,including large language models,are primarily train

207、ed on relatively simple tasks,such as predicting the next word(or part of a word)in a sequence of text.But these simple objectives have given rise to a vast array of practical capabilitiesmore than can be exhaustively tested for.73 Researchers continue to discover new methods to elicit significant p

208、erformance improvements that the models creators did not initially anticipate,with minimal additional training,through methods such as fine-tuning on tailored datasets,knowledge distillation from larger models,or prompting techniques such as chain-of-thought reasoning.74 Language models have even sh

209、own the ability to learn representations that extend beyond language tasks,proving useful for domains such as image classification and protein fold prediction.75 Even narrow models regularly demon-strate“transfer learning,”where knowledge gained from one task proves useful in others with varying deg

210、rees of relation.76 Implications for Catastrophic RiskAs AI systems proliferate,undetected latent capabil-ities could contribute to bad actors ability to initiate CNASDC14catastrophic events that the models developers may not have imagined possible.For example,researchers at the North Carolinabased

211、company Collaboration Pharmaceuticals inverted one AI tool designed for dis-covering therapeutic molecules as a thought experiment for a security conferenceand were surprised to find that within six hours,their inverted tool had proposed 40,000 candidate chemical compounds that might be viable chemi

212、cal weapons,including several known agents that were not included in the models training data.77 Though the adjustment was easily made,the researchers did not anticipate that their model could be so readily misused to such great effect.Similar incidents in other domains,especially where the AI model

213、s in question are publicly released,could expand the capabilities of bad actors.Technical Safety ChallengesTechnical safety challenges intrinsic to AI will continue to create vulnerabilities as AI tools increasingly integrate with sensitive systems.Though often arcane,tech-nical faults in AI systems

214、 can have dire consequences:for example,errors in image recognition systems in self-driving cars have already led to several fatalities.AI engineers are currently working to address these issues,though the degree to which they will ever be fully“solved”is an open question.As in many technical system

215、s,there will likely be incremental improvements to these issues that can always reemerge as AI systems develop more powerful capabilities and are applied in new contexts.ALIGNMENT AND SPECIFICATION GAMINGFor AI systems tasked with achieving particular goals,specifying objectives that accurately refl

216、ect their designers intentions remains a persistent challenge.78 Such systems have been known to find various ways of“hacking”the specified goals,often by violating unspec-ified or underspecified rules that might seem obvious or common sense to their programmers and are not explic-itly encoded into

217、the systems instructions.This is known as specification gaming.For example,one AI system instructed to win a boat race video game discovered that it could maximize its points by driving in chaotic circles through reward tokens rather than by completing the race.79 The effect is similar to how someon

218、e might exploit the letter of the law rather than following its spirit.Although the boat example is innocuous,as AI systems integrate with more complex systems,the consequences of specification gaming can become much more severe.Taken to the extreme,some fear that future,superintelli-gent AI systems

219、 misaligned with human interests could pose catastrophic or even existential risks.These risks are highly speculative,and expert opinions range widely M55 rockets containing the VX nerve agent are examined prior to their destruction in accordance with the Chemical Weapons Convention.VX was among the

220、 40,000 candidate compounds proposed by Collaboration Pharmaceuticals inverted AI system.(Program Executive Office,Assembled Chemical Weapons Alternatives via Flickr)15TECHNOLOGY&NATIONAL SECURITY|JUNE 2024Catalyzing Crisis:A Primer on Artificial Intelligence,Catastrophes,and National Securityabout

221、when or if so-called“artificial general intelligence”(AGI)or“superintelligence”could emerge,but the concern has gained traction among many in leading labs and some high-level political leaders.LOSS OF CONTROLOperators could lose control of AI systems for a variety of reasons,posing the risk of a“run

222、away”or rogue system causing damage in high-impact systems.Though this issue is often associated with aforementioned AGI or superintelligence concerns,it need not be:costly loss of control could occur in comparatively simple systems.80 Consider,for example,how the 2017 NotPetya cyberat-tack spread u

223、ncontrollably around the world at the cost of more than$10 billion,attacking systems in hospi-tals,global shipping companies,and factories.It even rebounded on the originating country,Russia,by hitting Rosneft,a state oil company.81 Emerging AI capabilities hold the potential for still more sophisti

224、cated autonomy,which could mean more dynamicand dangerousrisks from loss of control.ROBUSTNESSAI models are deemed“robust”when they consistently perform well across a wide range of conditions,espe-cially those that deviate from their training data sets.82 Achieving robustness can be challenging.Stra

225、tegies to enhance robustness can include diversification of training data;techniques to reduce overfitting to training data,such as ensemble systems that use multiple models in parallel to improve accuracy in their determinations;and provision for fallbacks(such as seeking human input)when encounter

226、ing anomalous situations.In some cases,performance may simply degrade in new contexts.But in others,AI systems may retain their capabilities while“misgeneralizing”their goalsor employing coherent strategies in pursuit of incorrect objectives.83 CALIBRATIONThe calibration of AI systems reflects how w

227、ell the confidence in their determinations corresponds to correctness.Calibration can help ensure that AI systems know when they can act confidently,and when to seek assistance or avoid high-stakes decision-making.84 The calibration performance of an AI model can be quantified by comparing its predi

228、ctions with outcomes.Measuring calibration can be more complex than it might initially appear,however,and high performance on one method of gauging calibration does not guarantee high performance on another.85 Calibration and robustness often go hand in hand,as calibration can be a particular proble

229、m in situations outside of training distributions.Conversely,well-calibrated models can help identify and mitigate the risks associated with poor performance in scenarios that deviate from training distributions.Implications for Catastrophic RiskIssues such as alignment,specification gaming,loss of

230、control,robustness,and calibration are all integral to ensuring that AI systems behave reliably and according to their intended purposes.To the degree that AI systems are used to help manage high-stakes processes,insuffi-cient attention to any one of these issues could contribute to catastrophic out

231、comes.A variety of military contexts could be applicable to these issues,most obviously if powerful lethal auton-omous weapons misfire under politically fraught conditions.Such a malfunction could catalyze political or military escalation with potentially catastrophic consequences,though such a cour

232、se of events would ultimately be determined by subsequent policy and strategy decisions.The U.S.military has been proactive in promoting rigorous standards for AI across its operations to avoid such scenarios,and it does not currently field lethal autonomous weapons systems that would initiate such

233、a chain of events.However,as lethal autonomous weapons become more sophisticated,the likelihood of consequential accidents or inadvertent escalation from technical AI challenges rises,particularly if rigorous standards are not adopted more widely.Weapons aside,AI systems entrusted to help manage hig

234、hly complex military logistics and maintenance systems could also have consequential impacts in the case of technical failures,in some circumstances potentially contributing to the chances of a catastrophe.Beyond militaries,AI systems used to help manage high-risk systems in nuclear energy,chemical

235、plants,bio-safety level 4(BSL-4)labs,cybersecurity,transportation systems,or elsewhere could also go awry in catastrophic ways due to technical flaws,and require appropriate mitigation measures.ADVERSARIAL ATTACKSAdversarial attacks can induce AI systems to err due to deliberately crafted malicious

236、inputs.These inputs are often designed to be imperceptible to humans,but with subtle changes that specifically target the AI systemfor example,adding a few pixels to an image of a cat to make it register as a dog.86 Adversarial manipulation can also be used to extract sensitive information from an A

237、I system or its training data.Foundation models face additional challenges in withstanding adversarial threats.LLMs,CNASDC16Emerging AI capabilities hold the potential for still more sophisticated autonomy,which could mean more dynamicand dangerousrisks from loss of control.for example,can be coaxed

238、 in plain English to produce outputs that contradict their safety training.Attacks can be even more powerful if the aggressors have influence over a models training.For instance,an attacker might insert“poisoned”data into a training set to make a model behave differently in certain situ-ationseither

239、 through actual infiltration,or through uploading information to the internet that might then be scraped by AI labs.There are currently no foolproof defenses against adversarial threats,especially without impacting the performance of AI models.Specific attack methods and defenses continue to develop

240、 in a cat-and-mouse game.87Implications for Catastrophic RiskOn a limited scale,researchers have already demon-strated how adversarial attacks can have dangerous effects in the real world.Two of the most notable examples include inducing an autonomous car to swerve into an oncoming-traffic lane thro

241、ugh carefully applied small markings on a roads surface,and using specialized glasses to spoof facial recognition security cameras and evade recognition or allow imperson-ation.88 Though these forms of adversarial attacks are unique to AI systems,many of their associated risks parallel those of cybe

242、rsecurity vulnerabilities:whether hacking conventional computer systems or hacking AI tools,both methods could in principle allow adver-saries to induce malfunction in strategic systems such as critical infrastructure.To the degree that high-im-pact systems such as critical infrastructure begin to u

243、se AI to help manage their complexity,so too will such systems be vulnerable to adversarial attacks.EXPLAINABILITY AND INTERPRETABILITY Advanced AI systems are increasingly built using deep learning models,which include many-layered“neural”networks with inner workings that can be very diffi-cult to

244、explain or interpret.89 As deep learning models become larger and their performance improves,the difficulty of understanding their inner workings becomes greater.A direct trade-off often occurs between performance and explainability,as models offer increasingly strong performance without devel-opers

245、 or users fully grasping how or why they make the decisions or outputs they do.90 Over time,some worry that continued reliance on highly usefulbut insuffi-ciently understoodmachines will lead to precarious accumulations of“intellectual debt”that can easily go awry as the difficulty of anticipating a

246、nd under-standing unexpected behavior compounds.91 Implications for Catastrophic RiskTechnical challenges in explainability and interpret-ability are unlikely to directly lead to a catastrophic event but have indirect relevance worthy of note.In instances of accidental technical malfunctions that mi

247、ght lend themselves to dangerous escalation,an inability to demonstrate how and why a system malfunctioned could exacerbate mistrust and accel-erate retaliatory action.Think,for example,of an AI-powered missile defense system erroneously firing on an adversarial nation.Conversely,if AI tools remain

248、largely inexplicable,their integration into an ever-wider set of national securityrelated systems would represent yet another arena for destabilizing gray zone operations,offering ample room for denial to cover subtle strategic attacks.For example,an adversarial actor could shut off access to crucia

249、l energy management systems by exploiting unknown vulnerabilities in an AI system helping to manage electricity grids.The adversary would have much greater leeway to claim that the AI system simply malfunctioned if the nature of its malfunction remained opaque.Likewise,if an AI system were known to

250、be opaque,an adversary could plausibly claim to be the source of a malfunction or to have the capability to cause a malfunction even if not true.Finally,AI systems lack of explainability makes it far more difficult to troubleshoot and address other technical issues reliably,posing long-term chal-len

251、ges to ensuring AI systems integrity,including in high-risk domains.Integrating AI into Complex SystemsIntegrating AI into complex systems presents an added layer of safety challenges that could have catastrophic effects.This aspect is often overlooked due to greater attention on the risks of new ca

252、pabilities and technical issues,but there is good reason to believe that how AI tools are integrated into broader systemsincluding how human operators respond to systems in practicewill be a key part of the risk profile for AI in high-risk 17TECHNOLOGY&NATIONAL SECURITY|JUNE 2024Catalyzing Crisis:A

253、Primer on Artificial Intelligence,Catastrophes,and National Securitydomains.92 Historically,mundane lapses of judgment and human operator errors have often been at the root of many automation-related tragedies,even if new technical capabilities and related technical flaws tend to dominate safety dis

254、cussions.In light of this reality,“human-ma-chine teaming”has emerged as a field of study to try to discern how the design of automation-enabled systems can best work with the cognitive and emotional par-ticulars of diverse operators.Moreover,even if there are no flaws with how users or operators en

255、gage with AI tools,the introduction of powerful new automation apparatuses into broader,complex ecosystems can,and often does,produce a range of unintended conse-quencesas with any transformative new technology.Given the seemingly limitless breadth of applications for AI tools,ensuring the safe depl

256、oyment of models requires,in each case,ensuring that the AI model is well suited to its operators and that the combination of the AI tool with other broader ecosystems does not generate unforeseen hazards.AUTOMATION BIASThe term“automation bias”refers to the tendency for individuals to excessively t

257、rust or rely on automated systems determinations,sometimes to the detriment of performance.93 This can occur even when the system clearly contradicts prior knowledge,intuitions,or training.In one study,for instance,participants who observed a robot perform poorly in a navigation guidance task noneth

258、eless all chose to follow the robot minutes later in a simulated emergency evacuation,including into a dark room with no discernable exits.94 OPERATOR TRUSTDespite the tendency for individuals to exhibit overcon-fidence in automated systems capabilities,there is also an opposite issue of ensuring th

259、at operators can maintain sufficient,appropriate trust in automated systems over time.A major emphasis of human-machine teaming research,ensuring appropriate amounts of operator trust for different types of AI-enabled systems,involves effectively communicating the systems capabilities and limits,as

260、well as how they perform under a range of dif-ferent circumstances and with different kinds of people or teams.95 As AI systems become more dynamic in their capabilities to execute complex tasks,the challenge of maintaining reliable,appropriate operator trust is likely to grow.96Implications for Cat

261、astrophic RiskAddressing issues associated with automation bias and operator trust is already a pressing issue in consequen-tial systems.The U.S.Army identified automation bias as a root cause of a pair of tragic missile misfires in 2003 related to target identification system errors,resulting in th

262、e deaths of two British lieutenants and an American lieutenant in two separate incidents.97 Though both cases are instances of friendly fire,it is not difficult to imagine a more dangerous scenario in which excess trust in a flawed automated weapons system could lead to an accidental attack on an ad

263、versary,catalyzing a cycle of rapid,violent escalation.Conversely,research by the Defense Advanced Research Projects Agency in collab-oration with Marines has already highlighted the high importance of building appropriate operator trust with autonomous military systems in advance of conducting oper

264、ations that would use such tools.98These issues extend far beyond military systems,however.Any operator of high-risk systems could be led astray by misplaced confidence in an automated systems erroneous determination,or could fail to effectively use high-impact systems due to insufficient trust in t

265、he system,to detrimental effect.As AI systems become more capable,the temptation to be overconfident in their determinations may grow for some applications,while maintaining sufficient confidence in their capabilities may be a challenge in others.THE LUMBERJACK EFFECTThe“lumberjack effect”suggests t

266、hat the more auto-mated a system becomes,the more difficult it is for human operators to effectively respond to system fail-ures.99 In other words,the higher the level of automation in a system,the harder it falls.An example of this is the 2012 Knight Capital trading accident,in which a flaw in high

267、ly automated trading software ultimately led to more than$460 million in losses to the firm.100 Despite relatively early detection,the complexity of the systems automation meant that its technicians needed more than 20 minutes to discover how to remedy the issue,a glacial speed in the algorithmic tr

268、ading world,and enough addi-tional time for the system to make a total of four million trades at tremendous cost.101ERODED SENSITIVITY TO OPERATIONSSafety theorists have identified“sensitivity to opera-tions”as one of five key traits that mark high reliability organizations(HROs),entities that have

269、been remark-ably effective in avoiding disasters.102 Sensitivity to operations means that operators maintain a real-time,integrated understanding of the full breadth of complex processes they are undertaking.As a result,they are able CNASDC18to quickly respond to anomalies and can more readily make

270、sense of unexpected situations.The introduction of automation into complex processes can predictably erode this sensitivity to operations by reducing the need for operators to actively engage with and monitor the processes and environment they are overseeing.103 This dynamic was one of the key cause

271、s of the 2009 Air France 447 tragedy,as the pilots reliance on automated flight systems reduced their sensitivity to the flights operations,setting the stage for the crash that killed all 228 passengers and crewmembers.In part stemming from the aircrafts automated systems,the pilots failed to fully

272、recognize the unusual environmental conditions in which they were flying.104 DESKILLING AND ENFEEBLEMENTAs AI systems take over an expanding range of func-tions and jobs that human operators once managed,the skills needed to manage those systems can atrophya process known as enfeeblement or deskilli

273、ng.105 There is some precedent for this:in the aforementioned 2009 Air France 447 crash,the French Civil Aviation Safety Investigation Authority cited an erosion of flight skills related to automation as a critical factor in the crash,because the pilots had insufficient experience navi-gating the un

274、usual conditions of their flight.According to analyses after the crash,the pilots would have likely gained these skills had they been trained on more flights that did not use such elaborate automation.106 Implications for Catastrophic RiskThe lumberjack effect,eroded sensitivity to operations,and de

275、skilling often overlap.Each degrades operators abilities to address problems related to AI systems as they inevitably emergewhether because of the com-plexity of the automation,reduced situational awareness,or atrophied skills to accomplish the automated task manually when necessary.All three tend t

276、o take root incrementally over time,as systems become more sophisticated and further remove operators from the operational environment,and technicians skills rust.The slow descent toward these problems makes them all the more insidious:at any one point,the extension of automation one step further in

277、 a process may make sense individually,but in aggregate can create environments of risk.Likewise,because the creep toward these issues is often slow and subtle,they are perhaps especially likely to affect high-risk systems when compared with other issues covered in this report.Whereas more obvious s

278、afety risks may receive considerable attention early on,these subtle,incremental challenges may only be noticed after it is too late.For this reason,as AI systems grow in reach and sophistication,engineers and system designers should be proactive in establishing practices French investigators inspec

279、t debris from Air France Flight 447 for clues on the causes of the tragedy.The investigators final report highlights both eroded sensitivity to operations and deskilling related to automation as key contributing factors.(Eric Cabanis/AFP via Getty Images)19TECHNOLOGY&NATIONAL SECURITY|JUNE 2024Catal

280、yzing Crisis:A Primer on Artificial Intelligence,Catastrophes,and National Securityand methods to mitigate these risks over time,including ensuring that functional analogue backup systems exist in safety-critical areas.Ensuring analogue redundancy,thereby reducing dependence on new technologies in c

281、ritical processes,is a practice that has been emphasized by cybersecurity experts for several years,with clear transferability to some AI applications.107 TIGHT COUPLINGTightly coupled systems are those in which constituent elements or processes within the system are directly and quickly responsive

282、to one another,leaving little room for adjustment or flexibility.Such systems may be necessary to accomplish certain tasks requiring high efficiency,but run the risk of having cascading effects if errors cause malfunctions.Because of the close interconnectedness of tightly coupled systems,such malfu

283、nctions can be very difficult to disentangle from one another.A paradig-matic example of tight coupling is the Three Mile Island accident,in which a rapid onset of confusing,interrelated failures obscured the root causes of malfunction related to loss of coolant in a nuclear reactor,resulting in a p

284、artial meltdown.Had the failures been less closely tied to one another in a system designed to give greater room and flexibility for intervention and oversight between pro-cesses,it may have been much easier to recognize and address the core issue earlier.108Implications for Catastrophic RiskAI-powe

285、red automation can lend itself to tight coupling in systems as AI promises to speed up virtually all processes that require attention to complex details.But excess tight coupling in high-risk systems could make catastrophic events more likely across a range of domains by reducing the resiliency of t

286、hese systems to errors,accelerating the impacts of errors across systems,and making the recog-nition of errors more difficult.Without careful attention to the dynamics of tight coupling,AI could threaten to exacerbate risks in high-impact systems across domains.EMERGENT BEHAVIOREmergent behavior ref

287、ers to unexpected behaviors or events that arise from the interactions between the parts of a complex system and its environmentespecially if the behavior or event cannot be easily reduced to the indi-vidual effects of those parts.109 To take a health example,if multiple medications are used to addr

288、ess multiple conditions in an individual,the intended effects of those medications might interact with one other in unex-pected ways to produce still further effects beyond what was intended.A classic example of AI-related emergent behavior is the case of the 2011 flash crash,in which an unknown num

289、ber of lightning-speed interactions between algo-rithms temporarily wiped out approximately$1 trillion in stocks in a matter of minutes.Though the event is believed to have been catalyzed by misleading market behavior from one individual,the extent of the damage was caused by the complex interaction

290、 of well-func-tioning algorithms playing off one another in ways that were simply not anticipated.110Implications for Catastrophic RiskThe integration of multiple AI tools into complex systems such as financial markets lends itself to more safety issues related to emergent behavior.111 The current c

291、hair of the SEC has warned that the introduc-tion of new AI tools into financial markets could lead to herding,a type of emergent behavior that can cause market instability and crashes.112 Finance and cyberse-curity are obvious candidates for emergent behavior risks,as both domains could host multip

292、le complex systems powered by AI tools that might play off one another in unexpected ways.But other domains could also be confronted with dangerous emergent behavior if multiple AI systems interact with one another,including weapons systems.RELEASE AND PROLIFERATION The way that AI tools are release

293、d and proliferate can shape AIs risk profile in a range of domains.These issues have sparked considerable debate in relation to foundation models like LLMs.Advocates of open-sourcing modelsmaking the underlying algorithms freely availableargue that open access to AI tools greatly accelerates the pro

294、gress of AI research and can act as a hedge against AI companies amassing too much power as the sole proprietors of powerful tools.113 Critics worry that such open release of AI models could pose serious risks,not least that latent,dangerous capabilities(see pages 1315)could be exploited by bad acto

295、rs in,for example,malicious hacking.In this view,once a model is open-sourced,the proliferation of that modeland its capabilitiesmay not be containable,and therefore providing models through“structured access”and asso-ciated safeguards is a preferable approach.114 Proponents counter that open-source

296、 approaches to many forms of software have helped improve their security and stability and may do so in the case of AI,and that open-sourcing models may also provide incentives to develop more thorough safety mechanisms in models for open release.115 This debate is ongoing.116CNASDC20But the issues

297、associated with the release and prolifer-ation of AI tools are also broader than the debate about open-sourcing foundation models alone,with commer-cial and scientific incentives shaping release strategies that vary between industries and domains.For example,how specialized dual-use scientific AI to

298、ols in biology are released represents another area of concern with significant implications for risks.Additionally,the formal release strategy of an AI tool does not always determine its proliferation.Meta,for example,intended to release one of its AI models only to researchers and civil society or

299、ganizations on a case-by-case basis,but the model was leaked publicly online after only a week.117 Hackers and states may also seek to steal powerful AI tools for their own ends.These fears have been especially pronounced in relation to China,which has a track record of stealing sensitive intellectu

300、al property in an effort to catch up to and surpass the United States technologically.In March 2024,a Chinese national was charged with stealing AI research trade secrets from Google.118 Implications for Catastrophic RiskAI tools with dangerous capabilities or hazardous technical deficiencies could

301、be released in ways that drive up risks,particularly if such dangerous capabil-ities proliferate widely.Combined with the fact that some capabilities are latent(see pages 1415)and that some hazards emerge only when AI tools are inte-grated into the broader environment,as detailed in the previous sec

302、tion,this heightens the relative importance of robust testing and evaluation capabilities to inform how AI tools should be appropriately released.But it also highlights the high priority of very strong security measures for developers that produce AI tools with potentially dangerous applications.Hac

303、king groups or states such as China,Russia,Iran,or North Korea may seek to gain unauthorized access to AI tools or informa-tion with dangerous applications,routing responsible release strategies altogetherand China has already demonstrated its proclivity to do so.Theft of AI tools by such actors,wit

304、h malicious intentions to use dangerous capabilities or without a thorough understanding of the dangers associated with an AI tool,could greatly exacer-bate the risks of misuse and accidents for such models.Conditions of AI Development The conditions of AI development will inflect all the preceding

305、dimensions of AI safetydetermining the time,attention,and resources that are devoted to these issues.Though often difficult to address directly given their systemic nature,issues related to the con-ditions of AI development are upstream of some safety challenges,and therefore represent some of the b

306、est opportunities for early intervention as catastrophic risks continue to take shape.Though corporate and geopolitical competition is often cited as the most prominent concern for ensuring safe-ty-friendly conditions of AI development,it is far from the only one worthy of attention.Safety cultures,

307、investment in safety research,social resilience,and engineers memory life cycles all play important parts in determining AIs catastrophic risk profile in the years ahead.CORPORATE AND GEOPOLITICAL COMPETITIVE PRESSURE As with technologies past,experts fear that com-petitive pressures among both AI c

308、ompanies and governments can quickly lead to security dilemmas and races to the bottom on safety.119 Periods of par-ticularly acute competition,such as heightened geopolitical tensions or aggressive commercial rival-ries,can further exacerbate the issue.These pressures tend to predispose AI companie

309、s and governments alike toward pursuing speed and power over precau-tions and safeguards where such tradeoffs exist.120 One example from industry is Ubers self-driving car unit,where a test vehicle struck and killed a pedestrian in 2018.Engineers disabled the vehicles emergency braking capabilities

310、in 2017,compelled by competitive pressures to provide a smoother rider experience.121 In the case of states,foreign competition was a significant contributing factor to the Chernobyl meltdown:Soviet leaders selected the flawed reactor design that enabled the tragedy in part due to its dis-tinctively

311、 Soviet developmentas opposed to designs that borrowed more from American schematicsand as a quick and cost-effective option for expanding nuclear energy in the Soviet sphere of influence to keep up with the United States ambitions to spread nuclear power in the wake of Eisenhowers“Atoms for Peace”s

312、peech.122Implications for Catastrophic RiskEscalating competitive pressures are already having effects on both corporate and state actors at the leading edge of AI development.123 Talk of an AI race between the United States and China is now commonplace,echoing some of the dynamics that characterize

313、d technology races between great powers in the past.One danger with historical precedent 21TECHNOLOGY&NATIONAL SECURITY|JUNE 2024Catalyzing Crisis:A Primer on Artificial Intelligence,Catastrophes,and National Securityis that inaccurate perceptions of adversaries capabili-ties or intentions can disto

314、rt safety priorities,which is all the more relevant today given the opaque nature of assessing and verifying AI capabilities compared with conventional military hardware.124 Appropriate caution also risks being sidelined if states fail to recognize that“superiority,”conceived of only in terms of cap

315、abilities,“is not synonymous with security.”125 Narrowly focusing on who is leading in AI competition in terms of technical capacity,without accounting holistically for the risks involved in developing and deploying powerful,high-risk capabilities,can miss the forest for the trees.At the same time,t

316、he United States cannot risk falling behind its adversaries in critical areas of AI development.This is especially true in regard to China,which has a stated goal of supplanting the United States as the world leader in AI by 2030.126 Given that companies in both countries are leading the technologys

317、 development,encouraging healthy corporate competition will likely be a strategic and economic priority for both nations,even as it can have adverse effects on safety.Policymakers and corporate leadership must walk a fine line in ensuring that they remain competitive,butto the extent possibleavoid t

318、he systemic safety pitfalls that often accompany competitive pressures in high-risk domains.Of course,safety and competitiveness are not always at cross-purposes,and can be mutually reinforc-ing.127 But there are good reasons for which competition is often cited as a primary contributor to concerns

319、about AI-enabled catastrophic risks.It is not difficult to imagine AI companies,with large profits on the line,cutting corners in safety to accelerate AI development for systems used in high-risk applications.Nor is it difficult to imagine countries militaries speeding AI adoption to keep pace with

320、one another in ways that exacerbate the risks of catastrophic accidents or miscalculations.DEFICIENT SAFETY CULTURESCultures of safety vary considerably among organiza-tions,industries,governments,and societies.Studies on HROs,for example,have demonstrated how a range of cultural traits in organizat

321、ions can greatly impact the likelihood of large-scale accidents,including preoccu-pation with failure,reluctance to simplify,sensitivity to operations,commitment to resilience,and deference to expertise.128 Certain industry-wide mentalitiessuch as those associated with Canadian-trained engineers,whe

322、re initiation traditions heavily stress safety and responsibilityhave also been noted to build greater safety awareness.This contrasts with other industries,for example in social media startups with their“move fast and break things”mentality that lends itself to reduced attention to safety.129 In te

323、rms of governments,autocracies are infamous for responding poorly to budding crises,often leading to catastrophic snowball effects.130 Finally,various societies exhibit a wide range of likelihoods for certain types of accidents and varying risk tolerances toward them,as indicated by differ-ences in

324、road traffic accident rates.131 As AI is developed and deployed in diverse contexts,these differences in safety cultures will inflect the safety and stability of the resulting systems.Safety cultures that are prone to acci-dents are more likely to have AI-related accidents and mismanage their effect

325、s.Implications for Catastrophic RiskConsidering the safety cultures in which AI tools are being developed and deployed is a significant,but often overlooked,priority for accurately assessing catastrophic risks associated with AI.While such an analysis is of relevance in a range of industry-and appli

326、cation-spe-cific cultures,Chinas AI sector is particularly worthy of attention and uniquely predisposed to exacerbate cata-strophic AI risks.132 Chinas funding incentives around scientific and technological advancement generally lend themselves to risky approaches to new technologies,and AI leaders

327、in China have long prided themselves on their governments large appetite for riskeven if there are more recent signs of some budding AI safety con-sciousness in the country.133 Chinas society is the most optimistic in the world on the benefits and risks of AI technology,according to a 2022 survey by

328、 the multina-tional market research firm Institut Public de Sondage dOpinion Secteur(Ipsos),despite the nations history of grisly industrial accidents and mismanaged crisesnot least its handling of COVID-19.134 The governments sprint to lead the world in AI by 2030 has unnerving resonances with prio

329、r grand,government-led attempts to accelerate industries that have ended in tragedy,as in the Great Leap Forward,the commercial satellite launch industry,and a variety of Belt and Road infrastructure projects.135 Chinas recent track record in other high-tech sectors,including space and biotech,also

330、suggests a much greater likelihood of catastrophic outcomes.136 Taken together,the AI-related catastrophic risks from China are particularly acute,with effects that could spread well beyond the country.SYSTEMIC UNDERINVESTMENT IN TECHNICAL SAFETY RESEARCH AND DEVELOPMENTAny combination of economic i

331、ncentives,underesti-mation of risks,or misaligned interests between the CNASDC22builders of AI and its users could lead to systemic under-investment in technical AI safety capabilities relative to overall capabilities,as some argue is already the case.137 Even though a number of leading AI labs have

332、 made safety research and development(R&D)a major priority,and the United States,United Kingdom,and Singapore have each established AI safety institutes,ensuring an appropriate balance between general AI capability research and safety research will be an ongoing challenge.138 Implications for Catast

333、rophic RiskOver time,a widening differential between general capa-bility development in AI and technical safety development creates conditions more conducive for catastrophes.If safety capabilities fail to grow commensurately with general capabilities,the allure of integrating AI into more complex,consequential systems will climb even as the ability to ensure the trustworthiness of those systems d