1、AUGUST 2024AI and the Evolution of Biological National Security RisksCapabilities,Thresholds,and InterventionsBill Drexel and Caleb WithersAbout the AuthorsBill Drexel is a fellow for the Technology and National Security Program at the Center for a New American Security(CNAS).His work focuses on Sin

2、o-American competition,artificial intelligence,and technology as an element of American grand strategy.Previously,Drexel worked on humanitarian innovation at the UN(International Organization for Migration)and on Indo-Pacific affairs at the American Enterprise Institute.Always seeking on-the-ground

3、exposure,Drexel has served as a rescue boat driver during Libyas migration crisis;conducted investigative research in the surveillance state of Xinjiang,China;and supported humanitarian data efforts across wartime Ukraine.He holds a BA from Yale University and masters degrees from Cambridge and Tsin

4、ghua universities.Caleb Withers is a research assistant for the Technology and National Security Program at CNAS.Before CNAS,he worked as a policy analyst for a variety of New Zealand government departments.He holds an MA in security studies from Georgetown University with a concentration in technol

5、ogy and security,and a bachelors of commerce from Victoria University of Wellington with majors in economics and information systems.About the Technology&National Security ProgramThe CNAS Technology and National Security Program explores the policy challenges associated with emerging technologies.A

6、key focus of the program is bringing together the technology and policy communities to better understand these challenges and together develop solutions.About the Artificial Intelligence Safety&Stability ProjectThe CNAS AI Safety&Stability Project is a multiyear,multiprogram effort that addresses th

7、e established and emerging risks associated with artificial intelligence.The work is focused on anticipating and mitigating catastrophic AI failures,improving the U.S.Department of Defenses processes for AI testing and evaluation,understanding and shaping opportunities for compute governance,underst

8、anding Chinese decision-making on AI and stability,and understanding Russian decision-making on AI and stability.AcknowledgmentsThe authors are grateful to Dr.Sonia Ben Ouagrham-Gormley and Dr.Nathan Paxton for their valuable feedback and suggestions on earlier drafts of this report,and to Michael A

9、ird,Samuel Curtis,and Christian Ruhl,who all provided further useful inputs.This report would not have been possible without contributions from our CNAS colleagues,including Paul Scharre,Vivek Chilukuri,Melody Cook,Rin Rothback,Allison Francis,Jake Penders,Tim Fist,Josh Wallin,Michael Depp,and Noah

10、Greene.The report was made possible with the generous support of Open Philanthropy.As a research and policy institution committed to the highest standards of organizational,intellectual,and personal integrity,CNAS maintains strict intellectual independence and sole editorial direction and control ov

11、er its ideas,projects,publications,events,and other research activities.CNAS does not take institutional positions on policy issues,and the content of CNAS publications reflects the views of their authors alone.In keeping with its mission and values,CNAS does not engage in lobbying activity and comp

12、lies fully with all applicable federal,state,and local laws.CNAS will not engage in any representational activities or advocacy on behalf of any entities or interests and,to the extent that the Center accepts funding from non-U.S.sources,its activities will be limited to bona fide scholastic,academi

13、c,and research-related activities,consistent with applicable federal law.The Center publicly acknowledges on its website annually all donors who contribute.TABLE OF CONTENTS01 Executive Summary03 Introduction04 The Current State of Catastrophic Biological Risks13 AI Safety and Biosecurity25 Capabili

14、ties to Monitor26 Recommendations30 ConclusionExecutive Summaryot long after COVID-19 gave the world a glimpse of the catastrophic potential of biological events,experts began warning that rapid advancements in artificial intelligence(AI)could augur a world of bioterrorism,unprecedented superviruses

15、,and novel targeted bioweapons.These dire warnings have risen to the highest levels of industry and government,from the CEOs of the worlds leading AI labs raising alarms about new technical capabilities for would-be bioterrorists,to Vice President Kamala Harriss concern that AI-enabled bioweapons“co

16、uld endanger the very existence of human-ity.”1 If true,such developments would expose the United States to unprecedented catastrophic threats well beyond COVID-19s scope of destruction.But assessing the degree to which these concerns are warrantedand what to do about themrequires weighing a range o

17、f complex factors,including:The history and current state of American biosecurity The diverse ways in which AI could alter existing biose-curity risks Which emerging technical AI capabilities would impact these risks Where interventions today are neededThis report considers these factors to provide

18、policy-makers with a broad understanding of the evolving intersection of AI and biotechnology,along with action-able recommendations to curb the worst risks to national security from biological threats.The sources of catastrophic biological risks are varied.Historically,policymakers have underapprec

19、iated the risks posed by the routine activities of well-intentioned scientists,even as the number of high-risk biosecurity labs and the frequency of dangerous incidentsperhaps including COVID-19 itselfcontinue to grow.State actors have traditionally been a source of considerable biosecurity risk,not

20、 least the Soviet Unions shockingly large bioweapons program.But the unwieldiness and imprecision of bioweapons has meant that states remain unlikely to field large-scale biological attacks in the near term,even though the U.S.State Department expresses concerns about the potential bioweapons capabi

21、lities of North Korea,Iran,Russia,and China.On the other hand,nonstate actorsincluding lone wolves,terrorists,and apocalyptic groupshave an unnerving track record of attempting biological attacks,but with limited success due to the intrinsic complexity of building and wielding such delicate capabili

22、ties.Today,fast-moving advancements in biotechnologyindependent of AI developmentsare changing many of these risks.A combination of new gene editing tech-niques,gene sequencing methods,and DNA synthesis tools is opening a new world of possibilities in synthetic biology for greater precision in genet

23、ic manipulation and,with it,a new world of risks from the development of powerful bioweapons and biological accidents alike.Cloud labs,which conduct experiments on others behalf,could enable nonstate actors by allowing them to outsource some of the experimental expertise that has historically acted

24、as a barrier to dangerous uses.Though most cloud labs screen orders for malicious activity,not all do,and the constellation of existing bioweapons norms,conventions,and safeguards leaves open a range of pathways for bad actors to make significant progress in acquiring viable bioweapons.But experts o

25、pinions on the overall state of U.S.biosecurity range widely,especially with regard to fears of nonstate actors fielding bioweapons.Those less concerned contend that even if viable paths to building bioweapons exist,the practicalities of constructing,storing,and disseminating them are far more compl

26、ex than most realize,with numerous potential points of failure that concerned parties either fail to recognize or underemphasize.They also point to a lack of a major bioattacks in recent decades,despite chronic warnings.A more pessimistic camp points to experiments that have demonstrated the seeming

27、 ease of successfully constructing powerful viruses using commercially avail-able inputs,and seemingly diminishing barriers to the knowledge and technical capabilities needed to create bioweapons.Less controversial is the insufficiency of U.S.biodefenses to adequately address large-scale biolog-ical

28、 threats,whether naturally occurring,accidental,or deliberate.Despite COVID-19s demonstration of the U.S.governments inability to contain the effects of a major outbreak,the nation has made limited progress in miti-gating the likelihood and potential harm of another,more dangerous biological catastr

29、ophe.New AI capabilities may reshape the risk landscape for biothreats in several ways.AI is enabling new capabilities that might,in theory,allow advanced actors to optimize bioweapons for more precise effects,such as targeting specific genetic groups or geographies.Though such capabilities remain s

30、peculative,if realized they would dramatically alter states incentives to use bioweapons for strategic ends.Instead of risking their own militaries or populations health with the unwieldy weapons,states could sabotage other nations food security or incapacitate enemies with public health crises from

31、 which they would be unlikely to rebound.Relatedly,the N1TECHNOLOGY&NATIONAL SECURITY|AUGUST 2024AI and the Evolution of Biological National Security Risks:Capabilities,Thresholds,and Interventionssame techniques could create superviruses optimized for transmissibility and lethality,which may consid

32、er-ably expand the destructive potential of bioweapons.Tempering these fears,however,are several technical challenges that scientists would need to overcomeif they can be solved at all.The most pressing concern for biological risks related to AI stems from tools that may soon be able to accel-erate

33、the procurement of biological agents by nonstate actors.Recent studies have suggested that foundation models may soon be able to help accelerate bad actors ability to acquire weaponizable biological agents,even if the degree to which these AI tools can currently help them remains marginal.2 Of parti

34、cular concern are AI systems budding abilities to help troubleshoot where experiments have gone wrong,speeding the design-build-test-learn feedback loop that is essential to developing working biological agents.If made more effective,emerging AI tools could provide a boon to would-be bioweapons crea

35、tors by more dynamically pro-viding some of the knowledge needed to produce and use bioweapons,though such actors would still face other significant hurdles to bioweapons development that are often underappreciated.AI could also impact biological risks in other ways.Technical faults in AI tools coul

36、d fail to constrain foundation models from relaying hazardous biological information to potential bad actors,or inadvertently encourage researchers to pursue promising medicinal agents with unexpected negative side effects.Using AI to create more advanced automated labs could expose these labs to ma

37、ny of the risks of automation that have historically plagued other complex automated systems,and make it easier for nonspecialists to concoct bio-logical agents(depending upon the safety mechanisms that automated labs institute).Finally,heavy investment in companies and nations seeking to capitalize

38、 on AIs potential for biotechnology could be creating competi-tion dynamics that prioritize speed over safety.These risks are particularly acute in relation to China,where a variety of other factors shaping the countrys biotech ecosystem also further escalate risks of costly accidents.Attempting to

39、predict exactly how and when cata-strophic risks at the intersection of biotechnology and AI will develop in the years ahead is a fools errand,given the inherent uncertainty about the scientific progress of both disciplines.Instead,this report identifies four areas of capabilities for experts and po

40、licymakers to monitor that will have the greatest impact on catastrophic risks related to AI:1.Foundation models ability to effectively provide experi-mental instructions for advanced biological applications2.Cloud labs and lab automations progress in dimin-ishing the demands of experimental experti

41、se in biotechnology3.Dual-use progress in research on host genetic suscepti-bility to infectious diseases4.Dual-use progress in precision engineering of viral pathogensCareful attention to these capabilities will help experts and policymakers stay ahead of evolving risks in the years to come.For now

42、,the following measures should be taken to curb emerging risks at the intersection of AI and biosecurity:Further strengthen screening mechanisms for cloud labs and other genetic synthesis providers Engage in regular,rigorous assessments of the biological capabilities of foundation models for the ful

43、l bioweapons lifecycle Invest in technical safety mechanisms that can curb the threats of foundation models,especially enhanced guard-rails for cloud-based access to AI tools,“unlearning”capabilities,and novel approaches to“information hazards”in model training Update government investment to furthe

44、r prioritize agility and flexibility in biodefense systems Long term,consider a licensing regime for a narrow set of biological design tools with potentially catastrophic capa-bilities,if such capabilities begin to materializeCNASDC2Introductionn 2020,COVID-19 brought the world to its knees,with nea

45、rly 29 million estimated deaths,acute social and political disruptions,and vast economic fallout.3 However,the events impact could have been far worse if the virus had been more lethal,more transmissible,or both.For decades,experts have warned that humanity is entering an era of potential catastroph

46、ic pandemics that would make COVID-19 appear mild in comparison.History is well acquainted with such instances,not least the 1918 Spanish Flu,the Black Death,and the Plague of Justinianeach of which would have dwarfed COVID-19s deaths if scaled to todays populations.4Equally concerning,many experts

47、have sounded alarms of possible deliberate bioattacks in the years ahead.There is some precedent:in the weeks following 9/11,letters containing deadly anthrax spores were mailed to U.S.lawmakers and media outlets,and the attack could have been considerably worse had the per-petrator devised a more e

48、ffective dispersion mechanism for the anthrax.The episode could portend a future in which more widely available biological capabilities mean malicious individuals and small groups devastate governments and societies through strategic biological attacks.Jeff Alstott,former director for technology and

49、 national security at the National Security Council,warned in September 2023 that the classified record contained“fairly recent close-ish calls”of nonstate actors attempting to use biological weapons with“stra-tegic scale.”5Accurately weighing just how credible such dire warnings are can feel next t

50、o impossible,and requires clear judgment in the face of opaque counterfactuals,alarmism,denialism,and horrific possibilities.But regardless of their likelihood,the destructive poten-tial of biological catastrophes is undeniably enormous:history is littered with examples of societies straining and ev

51、en collapsing under the weight of diseasesfrom ancient Athenss ruinous contagion during the Peloponnesian War,to the bubonic plague that crippled the Eastern Roman Empire in the 6th century,to the cataclysmic salmonella outbreak in the Aztec empire in the 16th century.6 It is essential that U.S.lead

52、ers soberly address the risks of biological catastrophewhich many claim will change dramatically in the age of artificial intelligence.Government and industry leaders have expressed grave concerns about the potential for AI to dramatically heighten the risks of catastrophic events in general,and bio

53、logical catastrophes in particular.7 In a July 2023 congressional hearing,Dario Amodei,CEO of leading AI lab Anthropic,stated that within two to three years,there was a“substantial risk”that AI tools would“greatly widen the range of actors with the technical capability to conduct a large-scale biolo

54、gical attack.”8 Former United Kingdom(UK)Prime Minister Rishi Sunak similarly expressed urgent concern that there may only be a“small window”of time before AI enables a step change in bio-terrorist capabilities.9 U.S.Vice President Kamala Harris warned of the threat of“AI-formulated bio-weapons that

55、 could endanger the lives of millions.and could endanger the very existence of humanity.”10 These are serious claims.If true,they represent a significant increase in bioterrorism risks.But are they true?This report aims to clearly assess AIs impact on the risks of biocatastrophe.It first considers t

56、he history and existing risk landscape in American biosecurity inde-pendent of AI disruptions.Drawing on a sister report,Catalyzing Crisis:A Primer on Artificial Intelligence,Catastrophes,and National Security,this study then considers how AI is impacting biorisks across four dimensions of AI safety

57、:new capabilities,technical chal-lenges,integration into complex systems,and conditions of AI development.11 Building on this analysis,the report identifies areas of future capability development that may substantially alter the risks of large-scale biological catastrophes worthy of monitoring as th

58、e technology continues to evolve.Finally,the report recommends actionable steps for policymakers to address current and near-term risks of biocatastrophes.While the theoretical potential for AI to expand the likelihood and impact of biological catastrophes is very large,to date AIs impacts on biolog

59、ical risks have been marginal.There is no way to know for certain if or when more severe risks will ultimately materialize,but careful monitoring of several capabilities at the nexus of AI and biotechnology can provide useful indications,including the effectiveness of experimental instructions from

60、foundation models,changing demands of tacit knowledge as lab automation increases,and dual-use AI-powered research into host genetic susceptibility to infectious diseases and precision pathogen engineering.Lest they be caught off guard,policymakers should act now to shore up Americas biodefenses for

61、 the age of AI by strengthening screening mechanisms for gene synthesis providers,regularly assessing the bioweapons capabilities of foundation models,investing in a range of technical AI safety mechanisms,and preparing to insti-tute licensing requirements for sophisticated biological design tools i

62、f they begin to approach potentially cata-strophic capabilities.I3TECHNOLOGY&NATIONAL SECURITY|AUGUST 2024AI and the Evolution of Biological National Security Risks:Capabilities,Thresholds,and InterventionsTThe Current State of Catastrophic Biological Riskso assess AIs emerging national security imp

63、acts on biological risks is difficult not only because of AIs unpredictable progress and varied applica-tions in the field,but also because simply establishing a clear baseline of biorisk today is a challenge.Perceptions of existing biorisks vary widely,as do the sources of potential threats.The fol

64、lowing sections provide an overview of the sources of catastrophic biorisk,evolving capabilities in biotech independent of AI tools,existing safeguards and gaps,and differing perceptions of risks.Taken together,significant,unaddressed biological risks to national security exist today independently o

65、f AI disruptions on the horizon,though some renditions of bioterrorist threats in particular are exaggerated due to an overly simplistic appreciation of the demands of bioweapons development.Sources of RiskThe COVID-19 pandemic shocked public consciousness into an active,daily awareness of biologica

66、l risks,an issue that previously was largely the purview of experts.Whether COVID-19 was the result of a lab leak or a natu-rally occurring event,the viruss vast disruptions and its likely death toll of nearly 29 million individuals consti-tuted a catastrophe of global proportions.12 For many in the

67、 biological sector who have been warning of the risks of pandemics,COVID-19 vindicated longstanding concerns over the vulnerability of both the United States and existing global response mechanisms to large-scale pandemics.Those concerns remain legitimate;in a highly connected world,conditions are r

68、ipe for devastating,fast-moving viruses to rapidly spreada risk also high-lighted,albeit less severely,by earlier pandemics such as severe acute respiratory syndrome(SARS)and swine flu.Future pandemics have the potential to be far worse.The 1918 Spanish Flu,for example,killed approximately 1 to 2 pe

69、rcent of the worlds populationequivalent to 70 to 150 million people today.Moreover,the 1918 pandemic had peak mortality for primeworking age adults,resulting in severe economic damage as millions succumbed to the illness.13 Before the discovery of modern antibiotics,bacterial pandemics such as plag

70、ue would sometimes kill half or more of affected populations.The Black Plague,for example,killed around half of Europeans over a few years in the mid-1300s.14Members of the St.Louis(Missouri)Red Cross Motor Corps on duty during the global influenza pandemic,October 1918.(Library of Congress)The comp

71、eting origin stories of COVID-19 both provide examples of catastrophic risk scenarios worthy of concern.Naturally occurring viruses can create catastrophes of devastating proportions independently of deliberate biological experimentation.Factors including increased travel,greater urbanization,climat

72、e change,changing interactions between humans and animals,and healthcare deficiencies in low-and middle-income countries all contribute to greater chances of extreme pandemics now and in the future.15But biological experimen-tation,too,can be a source of catastrophic risk.Controversial forms of scie

73、ntific research,such as gain-of-function research(also referred to as enhanced pandemic poten-tial pathogen research)that sometimes entails altering existing viral strains and creating new ones,have the potential to enable biological catastrophes by accident.16 The facts that the Wuhan Institute of

74、Virology engaged in gain-of-function research,acted as a center of coronavirus research,and elicited safety warnings within the U.S.Department of State before COVID all mean that a potential lab leak with catastrophic consequences CNASDC4was possible there,and could also be possible in a number of b

75、iological labs around the world.17 Indeed,a single bio-logical lab in Beijing was the source of four known SARS leaks in early 2004.18 Another lab in Lanzhou,China,leaked aerosolized Brucella to sur-rounding areas in 2019,leading to more than 10,000 individ-uals contracting the disease in what may b

76、e the largest lab leak to date.19 As of 2023,a total of 69 biosafety level 4(BSL4)laboratories world-widebiolaboratories that require the highest safety stan-dards to deal with extremely hazardous biological mate-rialsare in operation,under construction,or planned.20 In recent years,the number of su

77、ch high-risk labs has dramatically increased,with three-quarters located in urban areas.21 And between 1975 and 2016,there were more than 60 known accidents from lab researchers(BSL4 or otherwise)that resulted in individuals being exposed to highly infectious patho-genic agents.22 The true number,in

78、cluding unreported or unknown incidents,is likely much higher.23 Though many of these exposures did not come from BSL4 labs,and most were contained,the trend is not promising.Onlookers are often baffled by the growth of high-risk biological research,but there are strong incentives to engage in it.Pi

79、oneering remedies for particularly dangerous diseases or other biological agents is often associated with scientific prestige,can have financial benefits related to monetized cures,and can represent a lifesaving contribution to society.Whatever the motive,such research typically requires working wit

80、h,and some-times manipulating,biohazards.As such,some element of risk is ultimately unavoidable in experiments to advance medical understanding of high-risk pathogens.Security personnel stand guard outside the Wuhan Institute of Virology as members of the World Health Organization team investigating

81、 the origins of COVID-19 visit the institute in Chinas central Hubei province,February 3,2021.(Hector Retamal/AFP via Getty Images)Naturally occurring pandemics and lab leaks are not the only sources of risks for biological catastrophes.State and nonstate actors could create bioweapons with wide-ran

82、gingand potentially catastrophiceffects.Since the first documented use of biological weapons in the 14th century BCE,when the Hittites sent diseased rams to infect their enemies,armies have employed a wide range of now primitive tactics to use biological agents for strategic effects.These have inclu

83、ded infected or poisoned arrows and catapulting diseased corpses into besieged cities.24 A major turning point in the history of states use of biological weapons arrived at the end of the 19th century,as Louis Pasteur and Robert Koch provided the foundations for microbiology,thereby opening new poss

84、ibilities to understand and develop biological weapons.25 These new capabilities were explored in the first half of the 20th century by a range of countries,including France,the United Kingdom,Italy,Canada,Belgium,Poland,Germany,Japan,and the United States,and ultimately reached their apex in the So

85、viet Unions staggering bioweapons ecosystem.26 The Soviet Union created the largest bioweapons program in history,with roughly 15,000 scientists,technicians,and support staff directly working to produce hundreds of tons of biolog-ical weapons agentsincluding shocking efforts to make some of the worl

86、ds most deadly diseases more lethal and resistant to treatment.27Despite the Soviet Unions extraordinarily productive program,however,biological weapons remain relatively unattractive to most states due to their uncontrollable nature,except for limited specialized operations such as assassinations o

87、r special operations sabotage efforts.28 Since 1915,a total of 23 states have had known or sus-pected bioweapons programs,nearly all of which have been shuttered(if they existed at all).29 Japans World War IIera bioweapons program involved considerable 5TECHNOLOGY&NATIONAL SECURITY|AUGUST 2024AI and

88、 the Evolution of Biological National Security Risks:Capabilities,Thresholds,and Interventionsfatalities,including thousands who were killed for experimental purposes.30 Japans primary offensive deployments were executed in China,where Japanese forces reportedly poisoned more than 1,000 water wells

89、with cholera or typhus and distributed plague-in-fested fleas across several Chinese cities,among other activities.31 The 2002 International Symposium on the Crimes of Bacteriological Warfare,convened in China,estimated that the number of casualties from Japans bioweapons program in China amounted t

90、o 580,000 individuals at a minimum.32Though a range of governments have accused adversaries of deliberate state uses of bioweapons since Japans World War II operations,there is little evidence to substantiate most of these claims.33 A notable excep-tion is recently uncovered documentation suggesting

91、 that the Israeli military used typhoid and dysentery during the 1948 Arab-Israeli War,which it intended as a nonlethal means to deter Arab militiamen from returning to villages and towns they had been driven from and to impede the progress of invading Arab troops.34 Today,the U.S.State Departments

92、Bureau of Arms Control,Verification,and Compliance assesses that North Korea and Russia currently maintain offen-sive biological capabilities,while China and Iran engage in concerning biological research that may suggest they too maintain secret offensive capabilities of unknown size,or could quickl

93、y stand up such programs.35Notably,a state need not launch a biological attack for a catastrophe to occur.As with private scientific labs,state-run bioweapons labs can also have consequential accidental leaks,such as in the accidental 1979 outbreak of anthrax from the Soviet Unions Sverdlovsk lab.Th

94、e secret bioweapons facility emitted a plume of spores that were carried by wind over adjacent communities,killing 68 by official recordsthough the true number was likely greater.36 Another Soviet leak of highly lethal and trans-missible smallpox from a bioweapons research center on Vozrozhdeniye Is

95、land in 1971 could have been even more devastating,had it not been swiftly contained.37 Given that the products of bioweapons labs are dangerous by design,such leaks are comparatively more hazardous than those from most other scientific facilities.Terrorist groups or lone wolf individuals pursuing m

96、ass-casualty bioweapons also pose risks,though recent efforts of this kind have thankfully had limited impact.In recent memory,the United States was subjected to two bioweapon attacks,though neither reached cata-strophic proportions.In 1984,a religious commune in Oregon systematically contaminated l

97、ocal salad bars with salmonella in an attempt to incapacitate non-com-mune voters in a local election.In 2001,shortly after the 9/11 attacks,a suspected lone wolf perpetrator mailed letters filled with anthrax to news outlets in Florida and New York,as well as a congres-sional office building in Was

98、hington,DC,resulting in the deaths of 5 individuals and illness in 17 others.Both inci-dents could have been much worse.The orches-trators of the Oregon attack bought and considered using much more severe pathogens,but decided salmonella would be sufficient for their purposes.38 The powder used in t

99、he anthrax attack was only of a low grade and failed to disperse effectively,limiting its impact.39A hazmat worker sprays colleagues after an anthrax search at Dirksen Senate Office Building on November 18,2001,in Washington,DC.Authorities closed two Senate buildings to test for anthrax spores after

100、 investigators discovered a contaminated letter addressed to Sen.Patrick Leahy(D-VT).(Alex Wong via Getty Images)CNASDC6Devotees watch their leader drive by the Rajneeshpuram commune in 1982.Two years later,members of the commune contaminated local salad bars with salmonella to incapacitate non-comm

101、une voters in a local election.(Samvado Gunnar Kossatz)Some terrorist groups have held ambitions for still more far-reaching bioattacks that have thankfully not panned out,but could have been catastrophic.Al Qaeda and the Islamic State of Iraq and Syria(ISIS)have both sought mass-casualty biological

102、 weapons,as has the apocalyptic Japanese cult Aum Shinrikyo,infamous for its successful use of sarin gas,a chemical agent,to kill 13 individuals and injure more than 6,000 on the Tokyo Metro.40 These failures to effectively develop and deploy bioweapons reflect the delicate nature of biological agen

103、ts as opposed to chemical weapons or more conven-tional weapons.Taken together,the sources of potentially catastrophic biological risks span natural origins,accidents from legit-imate scientific experiments,and intentional production of biological agents from states,terrorist organizations,lone wolv

104、es,and apocalyptic groups.Each potential pro-genitor of biological catastrophe is subject to a distinct set of sometimes unpredictable forces and incentives that could alter the likelihood of developing dangerous pathogens or other bioweapons.Whereas state actors must respond to complex strategic in

105、centives and deter-rence dynamics,lone wolves and apocalyptic groups are largely impervious to such considerationswith terrorist groups operating somewhere in between.41 Even those attempting to develop bioweapons for strategic purposes can face challenges in managing them,raising the risk of potent

106、ially catastrophic incidents beyond their control,just as scientific motivations to explore dan-gerous diseases for medical progress can run the risk of costly accidents.But much as the motives and incentives around man-made biothreats vary,evolving technical capabilities help shape evolving risks f

107、rom each source.Evolving CapabilitiesThe risk profiles of biological catastrophesregardless of their sourceare heavily shaped by the capabilities of available biological tools and techniques.These capa-bilities can be thought of in two categories:“classic”or conventional biological capabilities,thos

108、e based on naturally occurring agents;and synthetic biological capabilities,those dependent on artificial manipulation of genetic code.A clear,if small-scale,example of a classic biological attack is the aforementioned 1984 salmonella attack in Oregon,organized by a religious commune that aimed at i

109、ncapacitating local voters in an election.42 Several naturally occurring biological agents,most notably anthrax and botulinum,could be leveraged to cata-strophic effect with the right dispersion mechanisms.TECHNOLOGY&NATIONAL SECURITY|AUGUST 2024AI and the Evolution of Biological National Security R

110、isks:Capabilities,Thresholds,and Interventions7Groups such as al Qaeda and Aum Shinrikyo,as well as the Soviet bioweapons program,have focused on these more accessible agents that could have plausibly led to mass-casualty events,albeit with greatly varying degrees of success.43 Though conventional b

111、iological capabilities have a more established history,leveraging naturally occurring biological agents still requires con-siderable expertise in terms of cultivating,sustaining,and dispersing biological agents effectively(though the difficulty of each of these areas,too,varies depending on which ag

112、ent is used).Synthetic biological capabilities rely on the tech-niques of synthetic biology more broadly,a field of research in which the genetic material of organisms is read,edited,and rewritten.Though synthetic biology has its origins in the 1970s,recently there has been con-siderable acceleratio

113、n in the speed,cost-effectiveness,and sophistication of synthetic biology.This increases the ease with which various actors can produce bespoke biohazards,as the Soviet Unions bioweapons program once aspired to do.The range of potential synthetic biological risks is potentially endless but could inc

114、lude altering or designing pathogens to be more lethal,more transmissible,less treatable,or less detectable.The proliferation of synthetic biology tools is accel-erating,primarily driven by a desire to revolutionize biology for the good of humanity.These tools have considerable potential to create a

115、 more robust bio-engineering ecosystem,which could facilitate rapid,collaborative breakthroughs in medicine,agriculture,CRISPR gene editing techniques,which allow greater precision in DNA editing than previously possible and are rapidly falling in cost.CRISPR enables biologists to manipulate pathoge

116、ns in a variety of new ways at relatively low cost.New DNA synthesis tools that allow scientists to order,combine,or“print”genomic sequences of organisms,including pathogens,with lower costs and increasing ease.Improvements in genome sequencing,which permit biologists to sequence DNA with increasing

117、 speed,accuracy,and cost-effectivenessan important element of testing and verifying potential biological agents.These three areas of technical advancement provide the foundation for a more dynamic biotech ecosystem but also augur new risks.One final element of the changing risks,for both conventiona

118、l and synthetic capabilities,is the rise of cloud labslaboratories that conduct biological pro-cesses and experiments on others behalf.Emerald Cloud Lab,for instance,describes itself as a“remotely operated research facility that handles all aspects of daily lab workmethod design,materials logistics,

119、sample preparation,instrument operation,data acquisition and analysis,troubleshooting,waste disposal,and everything in betweenwithout the user ever setting foot in the lab.”47 Since the first robotic biolab was launched 2012,biomanufacturing,and other applications.A key component of this push is“dem

120、oc-ratizing”access to advanced biological capabilities,which proponents hope will enable the field to achieve some of the fast-paced,collaborative success that has marked the software industry.44 But the flip side of democratizing access to con-structive synthetic biology applications is,perhaps,dem

121、ocratizing access to destruc-tive synthetic biology capabilities.Some of the new services,technologies,and tools associated with synthetic biology create a patchwork of capabilities that can be strung together in a range of combi-nations to produce biological agents or toxins with catastrophic poten

122、tial.45The biological tools and services powering advancements in synthetic bio-technology are complex and variated,but some key advancements include:46The Carnegie Mellon University Cloud Lab is a remotely operated,automated lab that gives researchers access to more than 200 pieces of scientific eq

123、uipment.(Carnegie Mellon University)CNASDC8that much harder to automate in robotic labs.Cloud labs have the advantage of being able to iterate on experi-ments more rapidly and over longer stretches of time than traditional lab technicians,allowing them to accumulate a repertoire of highly precise an

124、d replicable methods,even if their dexterity pales in comparison to conventional technicians.The compounding effects of these automa-tion techniques may add up,as more and more steps of experiments could be reliably strung together.That said,the degree to which this will be the case is ultimately un

125、known,and the reliable operation of such complex machinery and experimental operation may introduce its own forms of tacit knowledge that must be mastered,erecting new barriers to synthetic biology.Additionally,there are several important forms of tacit knowledgenot least those that relate to the in

126、tricacies of cooperation among technicianswith which automation will be very unlikely to help(for a more thorough exploration of the changing dynamics of tacit knowledge and automation,see“Tacit Knowledge,”page 21).51Ultimately,it may be too early to assess the impact of cloud labs on biorisks.Just

127、as such cloud labs could make bioweapon production easier for lone wolves,terrorist organizations,or apocalyptic groups,they could also centralize lab expertise under more controlledand monitorablebottlenecks of biological production.But given that not all cloud labs maintain robust safety monitorin

128、g systems for their orders,they could also raise the risks of malicious actors gaining access to potentially catastrophic biological capabilities.Safeguards and GapsThe primary safeguards against biological catastrophes of relevance to the United States include international orga-nizations that aim

129、to curb the deliberate production of high-risk biological agents;a variety of bodies,practices,and mechanisms designed to diminish the biological risks of experimental research;and the U.S.governments dedicated organs and law enforcement resources for bio-defense.Taking each in turn,the primary inte

130、rnational organizations of relevance include the following:The Biological Weapons Convention(BWC),a treaty that came into force in 1975,currently has 185 states parties that have committed to not produce or stockpile bioweapons,and to conduct only biodefense-related research in relation to bioweapon

131、s.With limited enforcement mechanisms and funding,the BWCs primary significance in biocatastrophe mitigation is in having helped establish an international norm with rel-atively few instances of noncompliance among states.Much as cloud labs aim to make biological experimen-tation faster and cheaper,

132、they also lower barriers to entry,including for potentially malicious parties.Rather than having to source biological lab equipment and cultivate the experimental skills that would have usually been required to develop biological agents,motivated actors could in principle outsource some or all of th

133、eir labora-tory needs to cloud labs,assuming they could circumvent cloud labs safety mechanisms.Nonetheless,there remain a number of hurdles to successfully automating sophisti-cated experiments(see“Tacit Knowledge,”page 21),and cloud labs maintain obvious incentives to not facilitate malicious acti

134、vities and develop their policies and safe-guards accordingly.48Even so,cloud labs ambitions to consistently automate an increasingly broad range of biological capabilities could represent a transformation in catastrophic bio-logical risks.As Dr.Sonia Ben Ouagrham-Gormley,an expert in the history of

135、 biological weapons,has demon-strated,experimental expertise and organizational culture have typically acted as the most significant inhib-itors to success for bioweapons production.From the Soviet bioweapons megaproject to the smaller,clandes-tine programs of Iraq,South Africa,and Aum Shinrikyo,att

136、empts to produce bioweapons have been most stymied by a combination of organizational challenges and gaps in“tacit knowledge”subtle expertise in the minutiae of experimentation that can be difficult or impossible to articulate.49 While the social and political hurdles that often accompany bioweapons

137、 programs may persist,cloud labs may alter the demands of tacit experimental knowledge by outsourcing some of the tacit knowledge needed to build biological agents.50The tacit knowledge in question can be as subtle as the air pressure in a lab chamber,the speed of swirling together a mixture,or minu

138、scule variations in the pH level of water used in an experiment.Often,it can be difficult for researchers to identify such variances in experimental conditions between labs,making the challenge seemingly rapidly growing companies such as Emerald Cloud Lab,Gingko Bioworks,and Synthego have offered se

139、rvices that reduce the need for biology professionals to manage the minutiae of conducting physical experiments themselves in favor of simply designing experiments for outsourced execution.Cloud labs represent an increasingly important element of the“digital-to-physical barrier,”through which digita

140、l designs or plans for biological production are made physical realities.By centralizing,standardizing,and auto-mating biolab resources and procedures,cloud labs aim to make considerable gains in life sciences efficiency and experimental reproducibility.9TECHNOLOGY&NATIONAL SECURITY|AUGUST 2024AI an

141、d the Evolution of Biological National Security Risks:Capabilities,Thresholds,and Interventions The Australia Working Group is an informal grouping of members of the BWC that meet annually to establish guidance on materials and tools worthy of export controls due to their ability to empower mali-cio

142、us actors.While this mechanism helps to constrain critical materials and tools that could exacerbate biorisks globally,there remain demonstrable gaps in these constraints that could be exploited.52 The International Gene Synthesis Consortium(IGSC)is a voluntary group of gene synthesis compa-nies,inc

143、luding cloud labs,that commit to screening both orders and customers for hazardous requests.While the consortium works to reduce risks,mali-cious actors can in theory still circumvent screening protocols to benefit from gene synthesis companies.53 Additionally,although nearly all of the worlds large

144、st companies making high-quality,gene-length DNA are members of the IGSC,there is at least one notable outlier in China,and many other labs and companies maintain more limited gene synthesis capabilities.54especially in regards to the oversight of research involving enhanced potential pandemic patho

145、gens.55 Moreover,some labs,especially private ones,may not be subject to the same oversight as government-funded or academic laboratories engaged in high-risk research.On occasion,some scientists may also simply circumvent ordinary oversight mecha-nisms,as with the 2018 He Jiankui incident in China,

146、in which Dr.He conducted illegal heritable human genome editing in three human fetuses.56Biodefense and preparedness measures provide a final set of safeguards against deliberate,accidental,and nat-urally occurring biological risks.Several governmental bodies work to establish wide-ranging biodefens

147、es,notably:The Biomedical Advanced Research and Development Authority(BARDA),which was estab-lished in 2006 in the Department of Health and Human Services to“develop medical countermeasures that address the public health and medical consequences of chemical,biological,radiological,and nuclear(CBRN)a

148、ccidents,incidents and attacks,pandemic influenza,and emerging infectious diseases”through a range of initiatives.57 The National Biosurveillance Integration Center,housed in the Department of Homeland Securitys Countering Weapons of Mass Destruction Office.It manages and analyzes important informat

149、ion about biological events among agencies to help enable better informed responses.A meeting of the Biological Weapons Convention(BWC)was held in Geneva in December 2014.The BWC is the primary multilateral agreement in effect to constrain the development,production,and stockpiling of biological wea

150、pons internationally.(Eric Bridie/U.S.Mission Geneva)In addition to these international bodies,various national practices including ethics review boards,biosafety commit-tees,and funding of due diligence mechanisms contribute to global biosecurity efforts.In the United States,such entities are infor

151、med by guidance from the National Institutes of Health.In the wake of COVID-19 and concerns over the American governments funding of potentially risky research,the risk tolerance that such mechanisms should exhibit has been a recent source of contention,CNASDC10CNASDC10CNASDC10 The Defense Threat Re

152、duction Agency in the Department of Defense,which works to deter,prevent,reduce,and counter weapons of mass destruction(WMDs)and emerging threats,including biothreats.These purpose-built entities aim to directly address biological catastrophic scenarios,and to complement the broader efforts of the C

153、enters for Disease Control and Prevention within the Department of Health and Human Services,which would take a leading role in addressing any biological catastrophe in the United States.National intelligence and law enforcement agencies also work to identify and prosecute actors who might try to bu

154、ild malicious bioweapons.But despite the wide range of organizations and institutions that work to mitigate biological threats to national securityand a reasonably clear picture of the sources and changing biotechnology capabilities that shape those threatsperceptions of the overall risks continue t

155、o vary considerably.Perceptions of RiskAssessments of the state of biological risks range considerably,most of all with regard to catastrophic bioterrorist threats.These worst-case scenarios rep-resent the most extreme cases that often animate the biorisk conversation in public discourse,and as such

156、 are a fitting place to start to establish the broader contours of the debate about contemporary biorisks.Much of experts divergences in opinion boil down to how they weigh different factors influence in the likelihood of threats emerging,including the availability of bio-weapons information,technol

157、ogical capabilities,and experimental experience.58 For those who believe cat-astrophic bioterrorism poses a severe risk,the rapidly advancing information and technology available to potential bad actors make worst-case scenarios increas-ingly plausible.Such scenarios,like the strategic release of a

158、highly lethal and contagious virus,could dwarf the impact of COVID-19.Others contend that the primary barrier to such worst-case scenarios is and always has been organizations challenges in wielding specialized experimental expertise,which continue to greatly constrain the potential of would-be biot

159、errorists today.This challenge,they contend,continues to significantly limit the capabilities of potential bioterrorists.This section considers each factor in turn,before exploring the comparatively less controversial issue of American vulnerability to catastrophic biological events,whether man-made

160、 or naturally occurring.Those who harbor strong bioterrorism concerns have warned for years that increasingly accessible biotech capabilities and widely available information on potentially catastrophic biological agents constitute a recipe for disaster.In this view,the United States is on borrowed

161、time:the absence of major incidents in recent decades owes more to luck than to effective risk management.By comparison with Aum Shinrikyo or the perpetrator of the 2001 anthrax attacks,actors today have access to far more powerful resources and readily available information on how to make biologica

162、l agents and on how biological weapons programs fail.In this view,new biological tools are easing the barriers to malicious actors building highly dangerous pathogens.Perhaps the clearest indication of this trend comes from a controversial experiment conducted in 2016,in which a private lab successf

163、ully constructed a horsepox virus from scratch by stitching together DNA segments that the lab legally purchased from a commer-cial company.At a cost of$100,000a larger sum than would be necessary todaythe lab was able to recreate from scratch a nearly extinct virus using entirely com-mercially avai

164、lable inputs.Troublingly,the horsepox virus is a cousin of the virus that causes smallpoxa disease that has been eradicated but that,if unleashed,would have catastrophic consequences due to its com-bination of high transmissibility and lethality,in a world of widespread lack of immunity.59Tellingly,

165、in the face of backlash against the exe-cution and publication of this research,the principal investigators primary defense was simply that there were no legal or informational barriers to conducting the experiment.Therefore,he maintained,the exper-iment itself did not substantially alter the risks

166、of a malicious actor constructing smallpox by the same methods.Some have disagreed,arguing that clari-fying the process and providing proof of concept are significant steps in the wrong direction.60 Regardless Much of experts divergences in opinion boil down to how they weigh different factors influ

167、ence in the likelihood of threats emerging,including the availability of bioweapons information,technological capabilities,and experimental experience.11TECHNOLOGY&NATIONAL SECURITY|AUGUST 2024AI and the Evolution of Biological National Security Risks:Capabilities,Thresholds,and InterventionsThis de

168、bate will likely continue to evolve as new biological tools emerge and proliferate.of how much the experiment did or did not impact overall risks,it at least demonstrated that the door to engineering powerful,smallpox-like viruses is open,at least as far as technical capabilities are concerned.To ma

169、ke matters worse,Jeff Alstott,former director for technology and national security at the National Security Council,warned in September 2023 that the classified record contained“fairly recent close-ish calls”of nonstate attempts to produce and scale bioweapons for strategic use,suggesting that the l

170、ike-lihood of groups attempting to field bioweapons may be more severe than some imagine.61 Taken together,proponents of this view argue that mounting risks of bioterrorism require immediate attention,lestsimilar to COVID-19experts warnings go unheeded,with catastrophic results.By contrast,those les

171、s concerned about recent technological advancements that might enable bad This camp also stresses that the only U.S.incident of note in the past quarter century,the 2001 anthrax attacks,had a highly limited impact,despite the high-level concern on these issues for several decades.63 If biological ri

172、sks were as grave as supposed,the thought goes,the dearth of significant large-scale bioterrorism events in the past century,both at home and abroad,suggests these risks may be exaggerateda symptom of overemphasizing technological capacity as the prin-cipal driver of risks to the exclusion of sociot

173、echnical factors such as experimental expertise and organiza-tional dynamics.64 From this perspective,more alarmist concerns may also be inflected by biosecurity experts incentives to stress worst-case scenarios,or,like experts in any field,an inflated sense of their works importance.There is undoub

174、tedly truth to both of these views:however quickly biotechnology tools are improving,the likelihood of would-be bioterrorists successfully actors point to the his-torical failures by states,terrorist organizations,apocalyptic groups,and lone wolves to create powerful biological agents.This track rec

175、ord,they argue,suggests that the risks are exaggerated,or at least misguided.Even if the theoretical knowledge and materials needed to make dangerous bioweapons are freely available,the barriers to producing working,effective biological agents at scale are very high,as any pharmaceutical company kno

176、ws well from the difficulty of developing and reliably producing biopharmaceuticals.Cultivating the organizational effectiveness and experimental exper-tise needed to develop,sustain,and deliver biological agents is extremely challenging,especially under the conditions of secrecy that bioterrorist a

177、ctors requirenot to mention the extreme sociological conditions under which extremist groups operate.And even if new technologies make biological tinkering more cost-effective and less onerous,learning to navigate new,delicate tools and systems that are constantly evolving tends to introduce new bar

178、riers to execution even as traditional ones diminish.62 By this logic,the aforementioned horsepox example is a case in point:that the lab in question had years of accumulated organizational and experimental experience enabling it to recreate the virus was more central to its success than was the ava

179、ilability of the tools and commercial resources that resulted in the virus.A terrorist group or lone wolf actor would struggle to do the same,as they have historically.fielding bioweapons will continue to be constrained by the complexities of getting secretive organizations to conduct extremely deli

180、cate experimental processes effectively.And while history shows that such coordi-nation and expertise are very difficult to achieve,it is also true that the availability of tools and information matters,and that dramatic changes to both in this sector will affect the ease with which nonstate actors

181、can develop and use bioweapons.This debate will likely continue to evolve as new biological tools emerge and proliferate.While useful to many actors,they may have outsized implications for the nonstate bioweapons threat.State and other advanced programs generally have greater existing capacity to wo

182、rk with hazardous biological agents,making the impact of new tools and information less dramatic.Consequently,discussions surrounding research accidents and state-level risks often revolve around familiar concerns like lab safety,regulations,and bioweapons policies.In contrast,more cantan-kerous deb

183、ates about bioterrorism threats often focus on the accessibility of new tools and sensitive informa-tion,and their relative impact on risks.But whether from malicious state or nonstate actors,scientific accidents,or naturally occurring diseases,there is growing agreement among many experts that the

184、United States current biodefenses are insufficient to effectively manage large-scale biological crises.65 To a degree,COVID-19 bore this concern out:though Operation Warp Speed was able to greatly accelerate CNASDC12the speed of vaccine development and delivery,efforts to contain the virus were larg

185、ely ineffective,in terms of both protecting the United States from contagion beyond its borders and within its society.If COVID-19 had been more lethal,or had a similar virus been stra-tegically deployed by an adversary,the United States likely would have suffered more severe losses in lives,economi

186、c vitality,and strategic maneuverability,with little ability to shape outcomes.Though there have been some efforts to address Americas vulnerability in the wake of COVID-19,notably the release of the U.S.National Biodefense Strategy and Implementation Plan in October 2022 and the establishment of th

187、e National Security Commission on Emerging Biotechnology,some experts fear that actions remain incommensurate with evolving risks.66 A range of organizations,including the Bipartisan Commission on Biodefense,the Nuclear Threat Initiative,and the Helena Institute,have advocated for the development of

188、 stronger measures to shore up American biological preparedness to cope with growing vulnerabilities from various potential originating sources.67AI Safety and Biosecurityo consider how AI will impact preexisting biological risks,this report draws upon its sister study,Catalyzing Crisis:A Primer on

189、Artificial Intelligence,Catastrophes,and National Security,which distills the literature on AI and catastrophic risks into four dimensions of AI safety of relevance to high-im-pact domains:68DimensionQuestionNew capabilitiesWhat dangers arise from new AI-enabled capabilities across different domains

190、?Technical safety challengesIn what ways can technical failures in AI-enabled systems escalate risks?Integrating AI into complex systemsHow can the integration of AI into high-risk systems disrupt or derail their operations?Conditions of AI developmentHow do the conditions under which AI tools are d

191、eveloped influence their safety?When applied to biosecurity,the most significant concerns around new capabilities center on the potential of AI-powered biological design tools(BDTs)to help develop more sophisticated biological weapons,and foundation models improving abilities to potentially help bad

192、 actors create bioweapons more easily.In terms of technical safety challenges,the related challenge of effectively constraining foundation models abilities to assist bad actors has dominated discussions so far,but there are other concerns worthy of note,including the development of AI tools with ill

193、-understood risks for therapeutics development.The integration of AI tools into broader biological systems could have a distinct set of impacts on risks,both from safety challenges that tend to emerge with automation,and related to the reduction of tacit knowledge barriers for less-experienced actor

194、s.Finally,corporate and geopolitical competitive pressures are exerting significant influence on the development of AI and biotechnology,and this could shape safety outcomes,particularly in China,where other conditions of biological and AI development exacerbate risks.New CapabilitiesEmerging AI cap

195、abilities hold tremendous promise for the biological sciences in two ways.First,AI tools are uniquely suited to turbocharge synthetic biology by providing novel,powerful means to interpret and manip-ulate genetic information toward specific ends.69 Though some of these capabilities are still matters

196、 of conjecture,there is good reason to think that AI holds tremendous potential to enable unprecedented advancements in biology and medicinewith significant implications for catastrophic risks.Second,AI foundation models may have the potential to amplify the biological capabilities of individuals wi

197、th limited biological knowledge or expertise.Though current AI tools effect in this area has been marginal so far,if such tools biological capabil-ities continue to improve,they may increase risks from nonstate actors,albeit with some important caveats.This report considers each of these AI capabili

198、ties in turn.AI AND BIOLOGICAL DESIGN TOOLSThough it has long been understood that human DNA encodes various genetic diseases and contributing factors to diseases,it remains beyond existing capabili-ties to fully understand the diversity of constellations of DNA segments at the root of different con

199、ditions.At 3.2 billion base pairs in length,and with vast variations from person to person,the human genome contains too much information for scientists using conventional methods to 13TECHNOLOGY&NATIONAL SECURITY|AUGUST 2024AI and the Evolution of Biological National Security Risks:Capabilities,Thr

200、esholds,and InterventionsTunderstand the precise dynamics of genes downstream effects.But such data-intensive,multivariable problems are precisely where AI excels:building complex models and detecting patterns and correlations across vast troves of data.AI holds tremendous potential to unlock unprec

201、-edented capabilities in the world of biologyin exploring not only the human genome,where much scientific research now focuses,but also the genetic material of pathogens and other organisms.Together with advances in CRISPR gene editing methods and gene sequencing technologies,AIs likely ability to d

202、iscern genetic patterns with greater preci-sion could act as a watershed development in synthetic biology,allowing unprecedented precision in manipu-lating genetic information toward deliberate goals.AI has already enabled significant advances in solving complex biological problems,such as in protei

203、n folding,where AI has reduced the time it takes researchers to understand many proteins shape from weeks or months to seconds,predicting structures with near-experimental accuracy.70Researchers have also used machine learning to help identify modifications to viral capsids(the protein shells of vir

204、uses)to better evade the immune system.This is a potential boon for gene therapy using non-pathogenic viral vectors,but it also raises concerns about the pos-sibility of transferring relevant knowledge or methods to the engineering of pathogenic viruses.71 Similarly,researchers have used machine lea

205、rning to develop models that predict the zoonotic and human-infectivity potential of viruses and pathogenicity of bacterial DNA,as well as mutations that help viruses such as SARS-CoV-2 overcome immunityof potential use for targeting anticipatory research and surveillance but also with obvious poten

206、tial for misuse.72The AI tools used to accomplish these feats are narrow systems referred to as biological design tools and will be at the heart of the coming biological revolution.73 Although at the moment BDTs abilities are still nascent,future BDTs may hold the potential for highly sophis-ticated

207、 design or editing functionalityincluding for pathogens.Editing the genetic material of pathogens to achieve particular effects is not new,but the potential of BDTs to accomplish this feat with greater precision could augur a step change in biological threats.Most concerningly,in principle it may be

208、 possible to design a more dangerous pathogen than has yet existed or that nature could produce on its own.Many biologists have suggested that there may be a naturally occurring evolutionary tradeoff between transmissibility and severity of diseases in naturally occurring pathogens.Some experts cont

209、est this hypothesis,and a host of factors and caveats complicate the idea,but reduc-tively it suggests that because viruses rely on living hosts to spread,natural selection tends to diminish the severity of the most lethal pathogens over time.74 Because a pathogen that is too severe will quickly die

210、 out,other less severe variants survive,multiply,and dominate.But if a BDT were used to modify or build a pathogen to optimize for lethality,transmissibility,and a long incubation period,in theory the resulting pathogen could transcend the natural pressures away from severity and result in a biologi

211、cal agent of unprec-edented destructive power.Such an AI-enabled“supervirus,”or anything approaching it,would constitute a risk of catastrophe of the highest order.Given that the BDT or,more likely,BDTs able to produce such a pathogen would have to be extremely advanced,it is likely that should such

212、 a capability arise,it would first be available only to highly advanced biolabs and state actors.However,only the most deranged actors intent on causing maximum uncontrolled destruction would be motivated to delib-erately create and release such a pathogen.Groups with such apocalyptic motivations ex

213、ist,for example Aum Shinrikyo,and sufficiently advanced BDTs could in principle enable small groups or even individuals to design such a supervirus.Thus,the potential for AI to greatly escalate worst-case scenarios in biological catastrophes istheoreticallyvast.However,advancing from current BDTs,su

214、ch as those used for protein design,to future BDTs that can edit pathogens to produce novel,specific effectsif such BDTs are possible at allinvolves complex steps.There would undoubtedly be significant hurdles to overcome.Ensuring that the bioweapon remains potent over time and through diverse geogr

215、aphies would also be a significant challenge,given how delicate pathogens often are.Similarly,it may be impossible to predict how such a pathogen would interact with human populations over time;it may regress to a less-lethal predominant variant.Even now,despite the considerable attention focused on

216、 COVID-19,virologists remain unable to reliably anticipate the impacts of new strains of the virus.75Another possibility for AI-powered BDTs to alter biological risks would be helping to design pathogens with more targeted effects in specific geographic areas or genetic populations.In principle,ther

217、e is reason to believe future BDTs with sufficient biodata could alter the horizons of possibility.Given that many viruses can only thrive under specific environmental condi-tions,such as temperature,humidity,and air pressure,CNASDC14dramatically escalate the incentives for states and other groups t

218、o develop and deploy bioweapons.If such capa-bilities do emerge,advanced state actors or advanced laboratories will most likely be the first to realize the potential of BDTs to create specialized biological agents.Given the degree to which BDTs could enable the most advanced biological actors to rai

219、se the ceiling of harm possible from biological agents,the introduction of advanced AI holds the potential to greatly expand the scope of biological catastrophic risks.However,these capabilities are still theoretical,and the timing and conditions under which such possibilities are realizedif everare

220、 exceedingly hard to predict,as is the extent to which BDTs will ultimately be able to achieve these prospective hazardous capabilities.Critically,uncertainty about the timing and extent of BDTs contributions to catastrophic risks cuts both ways:some risk scenarios may prove to be unfeasible entirel

221、y;but some may arrive much more suddenly than expected.In one study,for instance,an American pharmaceutical company found that it was unexpectedly able to use an AI tool usually used to help develop medicinal drugs to instead design new potential chemical weapons(see“The MegaSyn Experiment,”page 16)

222、.Similarly,some BDTs used for legitimate medical research could harbor unexpected hazardous capabilities that could be surfaced with minimal tinkering,though the greater complexity of biological agents compared to their chemical counter-parts may make this less likely.it stands to reason that if AI

223、can identify with greater precision exactly what elements of genetic information predispose viruses to environmental strengths and weak-nesses,it may be possible to optimize biological agents to work in particular locales.More disturbingly,given that different genetic populations have differing susc

224、epti-bility to some viruses,it may also be possible to optimize viruses to target specific populations or avoid others.Zhang Shibo,former president of Chinas National Defense University and former general in the Peoples Liberation Army,noted as early as 2017 that advanced biology techniques could en

225、able new offensive capabil-ities,including“specific ethnic genetic attacks.”76 There are also some less-obvious high-impact applications of such capabilities,including the creation of pathogens genetically targeted to induce crop failures in a countrys critical food supply chains,offering the potent

226、ial to stra-tegically disrupt adversaries food security.Like ultra-lethal superviruses,there remain many technical hurdles to overcome before viral engineering for geographic or genetic targeting is feasible.Arguably,such engineering is more tentative than superviruses,which have precedents in Sovie

227、t bioweapons and gain-of-function research,both of which made progress in enhancing the transmissibility or severity of viruses to humans without BDTs.There may also be unfore-seen limits or tradeoffs to just how precisely biological agents can be targeted to either geographic conditions or genetic

228、groups.Given that viruses mutate over time in unpredictable ways,ensuring that a BDT-engineered virus remains both potent and targetable would be a con-siderableand perhaps intractableproblem.Despite these caveats,if geographically or genetically targeted biological agents are ever achieved,the resu

229、lt will profoundly alter the incentives and deterrents to using bioweapons.For state actors,the imprecision of conventional biological agents has been the primary disincentive to employing them.A world in which such weapons could be targeted raises the specter of greater incentives to incapacitate e

230、nemy forces with a weapon that may be able to be administered with subtletyat least initiallyand to devastating effect.Though the most consequential impacts of BDTs remain theoretical,the threat they pose raises the poten-tial destructive capacity of biological agents dramatically from what was alre

231、ady a grave baseline.Existing bio-logical agents such as smallpox,anthrax,and botulinum hold the potential to catalyze catastrophes with millions of victims.Biological agents optimized toward even greater destruction could achieve exponentially more devastation.Similarly,precision bioweapons would F

232、OUNDATION MODELS AND DEMOCRATIZING RISKIf narrow-use BDTs hold the potential to dramatically escalate the impacts of biocatastrophes,general-purpose foundation models may also raise the likelihood of bio-catastrophes by helping to diffuse relevant expertise to a broader population.Foundation models

233、are AI tools trained on a large corpus of data to accomplish a broad range of tasks.79 Another common term is“frontier models,”which are highly capable general-purpose models that could pose considerable safety risks.80 Foundation models include large language models(LLMs)such as OpenAIs GPT-3.5,Met

234、as Llama 2,and Anthropics Claude 2,all of which can dynamically engage with users in natural language to communicate information,generate content,and even build websites and programs.Many leading AI labs are also building multimodal models,which can engage users not only with written text,but also w

235、ith photos,videos,and audio.The most widely used multimodal model today is OpenAIs GPT-4o,with which users on the web platform ChatGPT can engage via text,images,and audio.15TECHNOLOGY&NATIONAL SECURITY|AUGUST 2024AI and the Evolution of Biological National Security Risks:Capabilities,Thresholds,and

236、 InterventionsTHE MEGASYN EXPERIMENT:A GLIMPSE INTO FUTURE BDT RISKSpublic databases,are available with no oversight.”78 Notably,the AI model in question was not particularly complex;it ran on a 2015 MacBook.The MegaSyn experiment highlights practically how information technology is easing the acces

237、sibility of dangerous information,and the theoretical potential of BDTs to enable similar breakthroughs in biological weapons.AI tools such as MegaSyn can help avoid the toxicity of potential drugs and are an accelerant for more effective healthcare.But these tools are dual use.If they can avoid tox

238、icity,they can also pursue it.Given how focused the creators of these tools are on medicinal uses,they may not even recognize latent,weaponizable capabilities.The researchers behind MegaSyn,for example,were surprisedand alarmedby the ease with which their tool could generate toxic chemicals and thou

239、ght to attempt it only after being prompted to do so.If well-intentioned researchers in the future can use BDTs to alter virulence and transmissibility in genetic code,they may also inadvertently create novel capabilities to design more destructive pathogens.As in the MegaSyn case,repurposing AI sys

240、tems designed with good intentions into weapons may be an unnervingly easy proposition.Biological weapons are considerably more complex than chemical weapons,making the capabilities currently available in chemical applications only a loose analogy for prospective capabilities in biology.Additionally

241、,designing bioweapons is only one step in a much broader and more complex process of finding ways to produce,store,and disseminate viable bioweapons.Even so,the MegaSyn experiment provides a precedent for exactly the sort of risks that many fear from future,more capable BDTs.A graphical representati

242、on of a selection of molecules generated by MegaSyn(in salmon)and an existing dataset of toxic molecules(in turquoise),clustered according to their structural similarity(t-SNE,a statistical method,has been used to simplify the clustering to two dimensions).The chemical weapon VX,represented in purpl

243、e,was successfully generated by MegaSyn along with 40,000 other potentially viable chemical weapon candidates.Source:Urbina et al.,“Dual Use of Artificial Intelligence-Powered Drug Discovery.”As disturbing as the results of the study are,the unexpected ease with which the company was able to generat

244、e these results is perhaps even more unnerving.When a biosecurity conference invited the company to probe how drug discovery technology could be misused,Collaborations Pharmaceuticals researchers wondered if they could simply use their MegaSyn drug discovery AI model.Usually they used the tool to av

245、oid predicted toxicity in candidate drugs,but in this case they could instead optimize for it.By simply reversing the systems goal,and with virtually no AI engineering involved,the scientists were immediately able to generate 40,000 potential chemical weapons.They noted that many hundreds more compa

246、nies use similar AI tools for drug design,and that their“commercial tools,as well as open-source software tools and many datasets that populate Though the current risks of BDTs are limited,an experiment by a U.S.company offers an example of how AI can impact chemical weapons in a similar way to how

247、BDTs may eventually impact biological weapons.In 2022,researchers from Collaborations Pharmaceuticals,a North Carolinabased pharmaceutical firm,published findings from an experiment in which they used a drug design AI model called MegaSyn to design,in less than six hours,40,000 molecules with potent

248、ial for use as chemical weapons.Though not all the systems proposed molecules would work as chemical weapons,MegaSyn was able to successfully generate several agents known to be highly effective in that capacity,including VXone of the most toxic chemical agents known.There is also reason to believe

249、that the model pioneered entirely new classes of neurotoxins with previously unknown weapons potential.77CNASDC16Among general-purpose AI systems many capabilities is the ability to interactively distill scientific information,including biological information,into actionable steps to achieve particu

250、lar experimental results.While the capabilities of todays general-purpose AI systems are relatively limited in this regard,some experts fear that taken to their extreme,future,more capable foundation model systems could guide bad actors to build powerful biological agents.If so,making such tools wid

251、ely avail-able could dramatically expand the pool of individuals and groups able to cause a biological catastrophe.Some experts view the risks of foundation models helping bad actors develop a bioweapon as a pressing,even urgent,issue.Unlike sophisticated BDTs,gener-al-purpose large language models

252、already offer proof of concept in helping to accelerate dangerous biological activities,albeit with marginal,if any,benefits for success when compared with conventional internet assistance.Five recent experiments hint at the extent to which existing LLMs could accelerate bad actors acquisition of da

253、ngerous biological agents.First,in April 2023,researchers at Carnegie Mellon University created a system of interconnected LLMs thatwith access to the internet,code execution capa-bilities,hardware documentation,and remote control of an automated cloud laboratorywas able to achieve a surprising leve

254、l of experimental proficiency.The system was“capable of autonomously designing,planning,and executing complex scientific experiments”without human intervention,the most complex of which included suc-cessfully performing a cross-coupling reaction,a chemical process of several steps that would ordinar

255、ily require significant chemistry expertise.81 The system agreed to autonomously synthesize a common date rape drug and phosphene,a chemical weapon used in World War I.Only after a web search did the system refuse to synthesize other concerning compounds,including methamphet-amine;sarin;and VX,an ex

256、tremely toxic nerve agent(the system could autonomously search the internet to gain information).Though the experiment required consider-able technical expertise to create the system,the long-term ambition of many leading AI labs is to develop general-pur-pose foundation models with even greater cap

257、abilities in scientific experimentation.Likewise,though the results of this experiment were limited to chemical agentsgener-ally much simpler to produce than biological agentssome experts fear that rapid improvements in general-purpose AI systems in combination with rapidly improving biolog-ical clo

258、ud labs could mean that similarly powerful systems could soon be produced that are able to experiment with biological agents.Second,in a June 2023 paper,MIT researchers explored how LLMs might assist nonexperts in causing a pandemic by having nonscientist students use the models.In one hour,the LLMs

259、 proposed four potential pandemic-causing pathogens,explained how they could be created from synthetic DNA,suggested several DNA synthesis companies that were not likely to screen DNA orders,and proposed detailed protocols to assemble the pathogensincluding troubleshooting measures.The researchers a

260、rgued that the“results suggest that LLMs will make pandemic-class agents widely accessible as soon as they are credibly identified,even to people with little or no laboratory training.”82 Notably,the students were able to circumvent the safety measures in place on some of the platforms they were eng

261、aging with to access the sensitive information.Three important caveats mitigate these seemingly alarming results.First,having information about building pathogens is only the initial step in successfully building said pathogens.Additionally,one of the pathogens proposed was almost certainly too comp

262、lex to be feasibly produced by amateurs,and others may not pose much of a pandemic threat due to preex-isting immunity,even if they were achieved.83 And finally,while the information culled from the LLMs would have certainly accelerated bad actors progress,the same infor-mation is readily available

263、on the internet,though it might take more time to locate and distill into actionable plans.Results of a third experiment were published in July 2023,when Anthropic,a leading AI company,released an overview of an internal testing of their LLM for biological risks.Though they did not publish a detaile

264、d methodology,their study involved spending 150 hours with biosecurity experts to test their models ability to communicate biological information with poten-tially dangerous applications.They found that“current frontier models can sometimes produce sophisticated,accurate,useful,and detailed knowledg

265、e at an expert level,”though infrequently in most areas.84 They also suspected that“models gaining access to tools could advance their capabilities in biology.”85 Ultimately,this led the investigators to conclude that if left unmitigated,within two to three years,“LLMs could accelerate a bad actors

266、efforts to misuse biology relative to solely having internet access,and enable them to accomplish tasks they could not without an LLM.”86 Though this implies that current improvements over internet access are at most marginal,the authors noted that as general-pur-pose AI systems improve,their biolog

267、ical expertise also expands,suggesting that in the years ahead,AI systems will likely offer a substantive edge over traditional inter-net-based research.17TECHNOLOGY&NATIONAL SECURITY|AUGUST 2024AI and the Evolution of Biological National Security Risks:Capabilities,Thresholds,and InterventionsFourt

268、h,in January 2024,the RAND Corporation released the results of a study on the risks of large-scale biological attacks.To assess such risks,RAND tasked 14 small teams of researchers with devising operational biological attack plans.Each team was given a maximum of 80 hours of effort per team member o

269、ver the course of seven weeks to craft viable biological attack plans for large-scale effects.To evaluate the relative impact of LLMs,four control groups were given access only to the internet and were forbidden from using LLMs to augment their efforts.The resulting plans were graded by experts on b

270、oth operational and biological feasibility.The researchers found that there was“no statistically significant difference in the viability of plans generated with or without LLM assistance.”87 It should be noted that the tested LLMs included safeguards,meaning these results may not be applicable to us

271、ers given access to the raw models without safeguards(see“Technical AI Safety Challenges,”page 19).Fifth,OpenAI released results from a study in May 2024 examining whether GPT-4 could significantly enhance access to information necessary for creating bioweapons compared to internet access alone.The

272、study involved 100 participants,divided into expert and student cohorts,each randomly assigned to groups with either GPT-4 access or internet-only access.Researchers observed slight improve-ments in metrics such as accuracy and completeness,with experts experiencing approximately 0.9-point increases

273、 on a 10-point scale.When analyzing specific subtasks,no individual task showed statistically significant increases after controlling for multiple comparisons.88 However,the authors noted that if they had assessed the aggregate uplift in accuracy across all tasks,the result would have been significa

274、nt.89 Contrasting this with the RAND study,which reported an unambiguous negative result,the OpenAI researchers emphasized methodological differences,such as using a model without safety guardrails,a larger sample size,and varied task designssuggesting these factors might explain the discrepancy.90T

275、hese research effortsthe OpenAI study in partic-ularsuggest that while todays systems at most only marginally impact biological risks,foundation models may soon be able to improve the information available to malicious actors seeking to acquire bioweapons,especially nonstate groups.Such a developmen

276、t may be significant,but its effects can be easily underestimated or exaggerated,albeit for different reasons.To properly assess the import of LLMs potential for nonstate bio-weapons threats,it is best to consider them in the context of the previously mentioned debate about the perceptions of bioter

277、rorism risks(see“Perceptions of Risk,”page 11).To focus first on how the impacts of LLMs can be underestimated,it is important to appreciate that the information needed to develop a bioweapon is far more complex than a simple list of instructions.The value of foundation models in crafting bioweapons

278、 is not just in distilling complex biological information scattered across a wide range of sources into actionable steps,though that alone could represent a significant advantage over con-ventional methods that make use of the internet.Of equal importance is the budding ability of general-purpose AI

279、 systems to help triage how experimental processes have gone wrong,accelerating the design-build-test-learn feedback loop that is essential to developing working biological agents.91 To use a loose analogy,getting a good recipe for a rare,delicate baked good may be difficult,but even more important

280、is having an experienced chef to help inform how and why ones attempts at following the recipe did not go according to plan.Foundation models may hold the potential to eventually provide both recipes for bioweapons and expert advice on missteps in following those recipes.AI systems have already demo

281、n-strated an ability to successfully triage and correct where chemical experiments have gone wrong,suggesting a precedent for future biological experiments.92 As foun-dation models move from language to more multimodal capabilitiesable to interpret visual and other inputstheir ability to help triage

282、 where experiments have gone awry will likely grow.This enhanced assistance could be important across the entire bioweapons lifecycle,including the processes of effectively storing,sustaining,and deploying bioweapons.At the same time,however,there are often-over-looked limits to what sophisticated f

283、oundation models could contribute to nonstate actors biological produc-tion efforts.First,as previously mentioned,information availability is often overemphasized as an element of bioweapons production to the exclusion of other important factors that can pose barriers to success such as organization

284、al characteristics.93 Regardless of how dynamically foundation models are able to convey information to aspiring bioterrorists,the sociological conditions that frame their efforts may still make it exceedingly difficult to effectively use that informa-tion,especially across a multifaceted and length

285、y bioweapon lifecycle.Additionally,for the foreseeable future,foundation models simply will not be able to wield certain types of information that are often essential for successful experimentation.Some tacit knowledge,for example,is not articulated or absorbed verbally,and ordinarily must be passed

286、 on through apprenticeship or developed through accumulated CNASDC18Technical AI Safety ChallengesTechnical safety challenges intrinsic to AI tools could exacerbate biological risks in a variety of ways.The most obvious relates to foundation models potential to accelerate bioweapons production,as ex

287、plored in the previous section.To mitigate the chances of their systems being used for malicious purposes,AI devel-opers have attempted to create guardrails within AI systems that would prohibit their use for nefarious ends.But even while industry-leading foundation models are trained to refuse dang

288、erous or harmful requests,foolproof techniques to reliably constrain systems outputs have yet to be developed.With sufficient effort,models can be induced to bypass safe-guards through intentional crafting of“jailbreaks”or“adversarial prompts,”requests that are able to fool the system into ignoring

289、instructions or training to refuse to answer certain questions.95The difficulty in developing robust safeguards for advanced foundation models is related to the methods by which they are trainedusing gargantuan data sets culled from the internet and elsewhere.Because these datasets are so large,the

290、full extent of the knowledge that the models can acquire is difficult to identify,let alone control.The leading method to constrain outputs of information that creators would rather not be relayed is reinforcement learning from human feedback(RLHF).This technique iteratively fine-tunes a model based

291、 on human ratings of how its outputs meet the relevant policies and objectives of the creatorincluding not revealing harmful information.While this method works to a degree,it is a surface-level fix to the deeper issue.It reduces the models tendency to produce harmful results,but by and large does n

292、ot alter the fun-damental capabilities of the system,thus allowing clever tinkerers to find other ways to coax the desired illicit information from the model.While many advanced gen-eral-purpose AI system providers allow their tools to be accessed only behind an online user interfacein effect barrin

293、g users from tampering with the system itselfsome,such as Meta and Hugging Face,give users direct access to the raw models themselves.As researchers have demonstrated,users can then functionally undo the RLHF safeguards that would otherwise constrain the outputs of these models at low cost.96Researc

294、hers are working on more sophisticated ways to guide foundation models outputs using techniques to instill more robust guardrails.97 But as foundation models become more complex,including by integrating multi-modal data,it is possible that the difficulty of ensuring reliably controllable outputs may

295、 rise commensurately.98trial-and-error experience(for a more thorough explo-ration of these themes,see“Tacit Knowledge,”page 21).94 To stretch the baking metaphor,an experienced chef who can correct errors is much more helpful than a simple recipe,but if that chef is nonetheless unable to physically

296、 coach apprentices through the delicate,somatically intensive techniques required for success,success may never be fully achieved.Taken together,to the extent that foundation models prove to be a revolution in information assistance generally,they may also have the potential to provide a revolution

297、in information assistance for nonstate actors bioweapons development.But to the extent that actually developing and deploying bioweapons depends on several other indispensable factors,even a revolution in bioweapons information assistance is unlikely to directly equate to a revolution in bioterroris

298、ts capabilities.Finally,in addition to concerns about foundation models in isolation,it is also important to consider their evolving relationship with the risks of BDTs.As foundation models become more sophisticated,some experts believe that the issues associated with general-purpose AI systems and

299、BDTs may converge.This could occur either because general-purpose AI systems begin to acquire BDT-level expertise in synthetic biology in their own right,or,more likely,because they lower the barriers to using BDTs by acting as an interface that allows users to wield BDTs much more easily.In a worst

300、-case scenario,general-pur-pose foundation models may eventually compound the risks of BDTs,so that BDTs raise the ceiling on the destructive potential of biological catastrophes,while foundation models expand the number of individuals who can wield advanced BDTs or BDT-like capabilities.Such a scen

301、ario is far from certain,given the uncertain trajectory of both technologies,including new safety mechanisms that may emerge before such a conver-gence of risks.But to the extent that actually developing and deploying bioweapons depends on several other indispensable factors,even a revolution in bio

302、weapons information assistance is unlikely to directly equate to a revolution in bioterrorists capabilities.19TECHNOLOGY&NATIONAL SECURITY|AUGUST 2024AI and the Evolution of Biological National Security Risks:Capabilities,Thresholds,and InterventionsAccording to a range of organizational management

303、studies,increasing automation in complex systems can risk introducing a range of safety hazardsand biotechnology is no exception.The difficulty of constraining AI systems outputs has understandably dominated discussion of AI and biorisk,but a range of other general technical AI safety issues could h

304、ave impacts on biotechnology risks.These include explainability,over-and underfitting to training data,chal-lenges in generalizing beyond the training distribution,and failures to ground predictions with causal mecha-nisms and real-world physics.99 Such issues can certainly pose threats to the usefu

305、lness of systems,and can result in clinical dead ends or patient harm.But they are unlikely to contribute to full-scale catastrophic scenarios.One possible exception could be systems that are effective at identifying promising biological candidates for medical or therapeutic uses but unreliable in p

306、re-dicting the full impacts of such candidates.For example,in 2001 researchers inserted a gene for interleukin-4(a protein that supports immune response)into the mousepox virus,in hopes of finding a way to disrupt the reproductive systems of mice.But the resulting agent was an unexpected,highly leth

307、al variant of the virus that killed all the mice initially exposed,and subse-quently half of those vaccinated with an otherwise highly effective vaccine.100 In this case,humans had successfully pinpointed a seemingly promising research agenda but failed to anticipate the lethal end result.101 AI sys

308、tems with a similar blend of strengths and weak-nesses could make it easier for less-cautious actors to accidentally stumble upon dangerous biological agents,despite intending to develop medicinal products.While such incidents are not AI-specific,AIs ability to identify potential research paths at s

309、uperhuman speed,for reasons opaque to operators,could exacerbate such risks.Integrating AI Tools into Complex SystemsAs with other complex systems,such as the aviation industry or military systems,the integration of AI tools into the broader biotechnology ecosystem can reshape risks.In biotechnology

310、,these phenomena can be thought of in two broad categories.First,enhanced automation in a range of subfields can introduce new safety challenges through eroding technicians sen-sitivity to operations in their labs and disrupting the informal safeguards and incentives that accompany conventional lab

311、work.Additionally,AIs continued integration into biological processes may impact the role of tacit knowledge in acting as a barrier to bio-weapons production,though the degree of this impact remains to be seen.These issues arise from AIs ongoing,albeit nascent,transformation of experimenta-tion in f

312、ields such as gene editing and biomanufacturing,both part of a wider trend toward automation.102AUTOMATED PROCESSESAccording to a range of organizational management studies,increasing automation in complex systems can risk introducing a range of safety hazardsand biotechnology is no exception.103 Fo

313、r one,as biological experiments become more automated,researchers abilities to maintain robust sensitivity to operations of delicate and potentially dangerous processes could be adversely impacted.Research into high-reliability organizationsthose with remarkably strong safety recordsshows that the a

314、bility to maintain a compre-hensive,real-time understanding of the full complexity of an organizations ongoing operations is critical to avoid errors and accidents,and to ensure that mistakes do not snowball into catastrophes.104 In cases such as the U.S.Navys Submarine Safety Program,leaders have i

315、ntentionally designed their systems to ensure that human operators have active and deliberate over-sight of the most critical parts of processes for the sake of ensuring operators active awareness.105 New auto-mation capabilities in sensitive experimental processes in biology,without careful thought

316、 to how to maintain robust situational awareness among operators,also run the risk of eroding experimenters sensitivity to operations,potentially increasing the chances of lab accidents with catastrophic potential.AI tools and automation could also reduce the influence of suppliers and experts who t

317、raditionally provide formal and informal feedback and oversight to research activities.This shift could make research more efficient and accessible,but it could also make it more difficult to monitor and evaluate,and to intervene as risks or bad actors present themselves.Although suppliers and exper

318、ts ostensibly fulfill narrow tasks and processes within a bioscience ecosystem that could,in principle,be automated,the loss of their broader contextual awareness in performing tasks would be significant.It could mean that anomalies,irregularities,or suspicious behavior they might ordinarily notice

319、would go undetected by automated systems.CNASDC20Finally,automation could make certain lab procedures more precarious by disincentivizing safeguards and escalating the impacts of system failures.As automation enables more high-efficiency approaches to execute lab work at scale and biotechnology enga

320、ges in more industrial approaches to experimentation,safety checks may or may not scale appropriately to the expanding throughput.Additionally,stringing together multiple processes within larger automated procedures can result in“tightly coupled”systemsthat is,systems in which mistakes or failures i

321、n one area quickly spread to others as interdependent automated processes rapidly affect one another.In more conventional lab processes,scientists perform subtasks independently,with indi-viduals able to intervene in the case of failures from one task to the next.In more tightly coupled lab systems,

322、where scientists are less directly handling experimental tasks,flaws can have cascading or compounding effects as the process progresses.These automated processes often occur at machine speedsfaster than humans can reliably followexacerbating the risk.Much of the difficulty in addressing the issues

323、asso-ciated with automation stems from the fact that they can be exceedingly hard to identify,vary from lab to lab,and in many cases may not manifest at all.Additionally,it is not the case that the net effect of automation on safety is necessarily negative;to the contrary,in indus-tries such as avia

324、tion and healthcare,it has a significant safety-boosting effect.106 The same may be true of biology.Nonetheless,labs and other biotechnology service providers should pay close attention to the formal and informal safeguards that may be lost as they integrate greater automation into their systems,esp

325、ecially in terms of situational awareness,human oversight,and tightly coupled system processes.TACIT KNOWLEDGEHistorically,tacit knowledge has acted as a major bottleneck to successful biological development for scientists and bad actors alike.To understand how and why that is the case,it is importa

326、nt to recognize that tacit knowledge comes in many forms that are difficult to disseminate for different reasons.To that end,researchers James Revill and Catherine Jefferson explain tacit knowledge in terms of three broad catego-ries,with subcategories in each:107Weak tacit knowledge is that which i

327、n principle could be shared,but is difficult or impractical to communicate effectively between individuals or across organizations.For example,many organizations,including biolabs,employ logistically minded individuals who have such an intimate and wide-ranging understanding of the behind-the-scenes

328、 systems sustaining their orga-nizations operations that they can bring together information or resources to problem-solve in unique ways that can be lost if they leave the organization.Such knowledge can be difficult to recover without a replacement who has accumulated years of experi-ence in the o

329、rganizationeven when its custodians attempt to communicate itbecause the knowledge is experiential in nature.Another example is tacit knowledge that the bearer is unaware of having,so that in attempting to explain how to execute complex processes to others,he or she inadvertently fails to communicat

330、e information necessary to complete the task.Such“unrecognized”knowledge may seem simple enough to surface,but it can be exceedingly difficult to identify or articulate practices that have been subconsciously internalized,including in experi-mental processes.Somatic tacit knowledge refers to subtle

331、physical information that can be impossible to articulate and must be learned by doing,for example balancing on a bicycle.Such“muscle memory”information can be critically important to successful biological experi-mentation and can be as subtle as different techniques to swirl,pipette,or pour solutio

332、ns,but also includes highly delicate lab procedures that require specialized techniques.108 Knowledge such as this often cannot be disseminated without in-person instruction,a fact that has been demonstrated time and again in“do-it-yourself”biology communities experiences trying to conduct their own

333、 experiments.109Communal tacit knowledge refers to collective expertise produced and accumulated by teams of specialists working together over time.Just as sports teams practice together repeatedly to develop plays,tactics,and rhythms that make effective use of each players skills and roles,so too experts with diverse technical skills and roles must learn to combine their efforts effectively throu