新興技術與安全中心：2023年生成式人工智能的迅速崛起評估安全與風險（英文版）（86頁）.pdf

《新興技術與安全中心：2023年生成式人工智能的迅速崛起評估安全與風險（英文版）（86頁）.pdf》由會員分享，可在線閱讀，更多相關《新興技術與安全中心：2023年生成式人工智能的迅速崛起評估安全與風險（英文版）（86頁）.pdf（86頁珍藏版）》請在三個皮匠報告上搜索。

1、0 The Rapid Rise of Generative AI Assessing risks to safety and security Ardi Janjeva,Alexander Harris,Sarah Mercer,Alexander Kasprzyk and Anna Gausen December 2023 RESEARCH REPORT Ardi Janjeva,Alexander Harris,Sarah Mercer,Alexander Kasprzyk and Anna Gausen 1 About CETaS.2 Acknowledgements.2 Execut

2、ive Summary.3 Recommendations.8 1.An Introduction to Generative AI.11 1.1 A short history of AI.11 1.2 The pace of change.13 1.3 Methodology.17 2.Evaluating Political,Digital and Physical Security Risks.20 2.1 Political security.21 2.2 Digital security.28 2.3 Physical security.33 2.4 Weighing malici

3、ous and incidental sources of risk.36 3.Generative AI and Future Intelligence Capabilities:Opportunities and Limitations.43 3.1 Enlarging the investigative toolbox:analysis and summarisation.43 3.2 Open-source and commercial models:innovation vs risk management.52 4.Governance,Policy and Regulation.

4、55 4.1 Signalling and reporting.56 4.2 Prohibition and red lines.66 4.3 Strengths and weaknesses in the legislative environment.67 4.4 Global governance.68 4.5 Training,guidance and safeguards.74 Case Studies.77 Case study 1:OSINT Summarisation.77 Case study 2:Synthetic Data Generation.80 About the

5、Authors.84 The image used on the cover and back cover was generated by OpenAIs DALL-E 2.The Rapid Rise of Generative AI:Assessing risks to safety and security 2 About CETaS The Centre for Emerging Technology and Security(CETaS)is a research centre based at The Alan Turing Institute,the UKs national

6、institute for data science and artificial intelligence.The Centres mission is to inform UK security policy through evidence-based,interdisciplinary research on emerging technology issues.Connect with CETaS at cetas.turing.ac.uk.This research was supported by The Alan Turing Institutes Defence and Se

7、curity Programme.All views expressed in this report are those of the authors and do not necessarily represent the views of The Alan Turing Institute or any other organisation.Acknowledgements The authors wish to thank all those who took part in a research interview or focus group for this project;th

8、ey are especially grateful to Alena Frankel,Bertie Vidgen,David C,Robert C,and Sam Stockwell for their valuable feedback on earlier versions of this report.Ardi Janjeva,Alexander Harris,Sarah Mercer,Alexander Kasprzyk and Anna Gausen 3 Executive Summary This CETaS Research Report examines the implic

9、ations of generative AI for national security.The findings and recommendations are based on a combination of openly available literature and research interviews with more than 50 experts across government,academia,industry,and civil society.To our knowledge,this represents the most comprehensive pub

10、licly available UK-based study on the national security implications of generative AI.Generative AI is a form of AI that can generate content such as images,audio,and text based on user suggestions.The multitude of possible generative AI use cases is seen by some as an opportunity to revolutionise t

11、he way that individuals interact,and businesses operate.However,from a national security perspective,the forms in which generative AI augments human productivity represent a significant challenge and typify the way that technology is continually stretching the boundaries of national security.The sec

12、urity risks posed by generative AI may be understood as either augmenting pre-existing societal risks or as posing completely novel risks.In most cases,generative AI lends itself to the former:security risks like disinformation,fraud,and child sexual abuse material are not novel creations of generat

13、ive AI but are amplified in speed and scale by the technology such that they may harm a larger proportion of the population than before.Understanding the national security picture in this way should dampen unwarranted hysteria regarding the unprecedented nature of the threats posed,while enabling a

14、more targeted focus on the threat areas where generative AI catalyses risk.Generative AI also offers potential opportunities for use within the national security community.Currently,generative AI tools are too unreliable and error-prone to be trusted in the highest stakes contexts within national se

15、curity.This means they are not ready for use where they are required to make a decision or where explainability is required to satisfy accountability and oversight requirements.For those who may want to use generative AI to undermine UK national security,inaccuracy is less important if a large langu

16、age model(LLM)underperforms in its generation of deepfakes or in writing malware,the cost of failure to the attacker remains low.But from a defensive perspective,similar errors could lead to significant security breaches.Users propensity to overly trust LLMs might lead to a reluctance to challenge A

17、I-generated outputs.The national security and technology discourse has historically focused on understanding threats from adversaries;groups or individuals who set out to inflict harm.However,the proliferation of advanced technology to a much wider constituency calls for a shift in mindset to accoun

18、t for all the unintentional or incidental ways in which generative AI can pose national security risks.This is seen in the range of possible instances of improper AI The Rapid Rise of Generative AI:Assessing risks to safety and security 4 adoption defined as the inappropriate and misguided attainmen

19、t and deployment of AI systems.In contexts including critical national infrastructure(CNI),public services,the private sector,and individual DIY experimentation,the fear of missing out on the crest of the generative AI wave may cloud judgments about higher risk use cases.For explicitly malicious gen

20、erative AI use cases,threats can be understood as falling into one of the three categories of digital,physical,and political security.Digital security Physical security Political security Cybersecurity By reducing the degree of specialist knowledge required,generative AI can assist the less technica

21、lly able user in experimenting with novel cyberattack techniques and increasing their sophistication iteratively.Less certain is whether generative AI will enable wholly new types of cyberattack that even the best attackers would not previously have been aware of the most significant longer-term con

22、cern from a national security perspective.Radicalisation and terrorism The personalised relationships that individuals can now form with AI chatbots,in part due to their constant availability and limitless patience,could alter the radicalisation blueprint.However,there remains a distinctly human ele

23、ment to this process which the current generation of generative AI will be unlikely to replicate soon.Specificity about which stages of the terrorist enterprise generative AI is likely to augment is important for some groups generative AI may be more useful for glorification than radicalisation.Poli

24、tical disinformation and electoral interference Generative AI could be a force multiplier for political disinformation.The cumulative effect of generative text,image,video,and audio will exceed the impact that any one of those modalities can have individually.Scale could be significantly enhanced by

25、 improvements in usability,reliability,and cost-effectiveness of LLMs,while personalisation could reach new levels of convincingness with more impressive storytelling and individually tailored campaigns.In the hours or days preceding an election,it would be challenging to identify and discredit a ma

26、licious AI-enabled information operation.Targeting and fraud Fraudsters stand to benefit significantly from generative AI.Qualitatively,generative AI can assist fraudsters with more professional-looking,highly targeted spear phishing,Weapon instruction The generation of publicly accessible but hard-

27、to-find information decreases the degrees of separation from information pivotal to developing and executing an Surveillance,monitoring,and geopolitical fragmentation Generative AI could play an important role in furthering the global proliferation of technology which adheres to authoritarian standa

28、rds and Ardi Janjeva,Alexander Harris,Sarah Mercer,Alexander Kasprzyk and Anna Gausen 5 increasing the burden of resilience on potential victims.Quantitatively,developments in autonomous agent frameworks could enable wide scale automation of fraud attempts.Improvements in the domain of voice cloning

29、 are an area of particular concern in the fraud context.attack plan.This risk is exacerbated if web APIs permit the connection of large pretrained models into physical systems,which are allowed to take direct actions in the world.Nonetheless,in the biochemical weapons context,there is a significant

30、technical leap from prompting a chatbot to synthesising lethal materials,which limits the utility of generative AI for low-skill actors.values,aiding attempts to enforce single versions of historical truth for future generations.Democracies may be more vulnerable to the exploitation of the creative

31、characteristics of generative AI systems than autocracies.This emphasises the need to understand the cultural and behavioural aspects to generative AI use around the world.Child sexual abuse material The proliferation of AI-generated CSAM is a significant concern for law enforcement agencies.The dif

32、ficulty of distinguishing real from fake images will continue to increase and pose the challenge of false negatives slipping through the net.At the same time,there is a false positive risk where law enforcement investigates images created of children who have not been physically abused,diverting sca

33、rce resources away from those who have.Despite their unreliability in very high stakes national security contexts,generative AI does offer various opportunities for national security and law enforcement agencies.In the intelligence analysis context,the role of generative AI is best understood as enh

34、ancing individual productivity.Using generative AI as cognitive co-pilots across the direction,collection,processing,and dissemination stages of the intelligence cycle could alleviate traditional challenges regarding the fact-poor and opinion-rich environment analysts operate within.Nonetheless,care

35、ful deployment involving frequent human validation is crucial at this early stage of maturity and familiarity.Autonomous agents artificial entities that can sense their environment,make decisions,and take actions without human intervention could be an accelerating force within the The Rapid Rise of

36、Generative AI:Assessing risks to safety and security 6 intelligence and security context,due to their ability to draw on other data sources for additional validation.In theory,teams of agents could be used to rapidly process vast amounts of open-source data,provide preliminary risk assessments,and g

37、enerate hypotheses for human analysts to explore further.However,until the underlying LLMs can provide reliable(consistent,correct,and safe)and accurate responses,agents will be at risk of delivering unpredictable or misaligned outcomes.The key mitigations in addressing these challenges are accounta

38、bility,transparency,and human oversight of both the actions taken by the agent and the inference performed by the system.To respond to the complex landscape outlined above,governments must devise policy interventions which have three main goals:to create better visibility and understanding of genera

39、tive AI systems;to promote best practices;and to establish incentives and enforcement of regulation.Establishing signalling and reporting mechanisms into government and relevant third-party actors,and red lines in the highest-risk contexts(such as decision-making within critical national infrastruct

40、ure)are important aspects of achieving these goals.Signalling Reporting Red lines Watermarking Automatically adding labels or invisible watermarks to AI-generated content is a possible technical solution to the challenges of generative AI-enabled disinformation.However,concerns persist over its vuln

41、erability to deliberate tampering and the ability of bad-faith actors to bypass it entirely.Disclosure and explainability The challenges associated with AI detection tools place additional emphasis on disclosing when generative AI is being used,and issuing clear guidance on appropriate use and warni

42、ngs for misuse.Better outcomes will be co-dependent on the level of explainability provided by the system and individuals ability to interpret AI outputs.Multi-layered and socio-technical evaluation To understand the full spectrum of national security implications,AI system evaluation must go beyond

43、 the capabilities of any individual model.A multi-layered,socio-technical approach to system evaluation is needed to understand how human interactions and systemic factors interact with technical components of generative models to amplify different types of risks.Release strategies Rapid increases i

44、n capability can mean policymakers are ill-prepared for the next game changing innovation.Leading AI developers recently committed to avoid releasing models without prior testing by government agencies,but this process must be open and transparent enough to ensure public trust in its conclusions.Pre

45、-empting the high-stakes contexts where generative AI should not be used will prevent situations where the technology can take irreversible actions without direct human oversight or authorisation.Ardi Janjeva,Alexander Harris,Sarah Mercer,Alexander Kasprzyk and Anna Gausen 7 Writing of this report c

46、oincided with the UKs AI Safety Summit in November 2023,and contemporaneous announcement of a new Government-sponsored AI Safety Institute for safety testing of the most advanced AI models.The coming months will be crucial in determining the role and scope of the new AI Safety Institute,and the UKs

47、approach to managing emerging AI risks more broadly.At the international level,there are two key actions the UK could take to narrow existing disparities between governance models:promoting shared evaluation tools and clear targets;and contributing to international regulatory expertise and capacity.

48、The announcement of the new AI Safety Institute is a positive step in this regard,but the UK must make leaps in the three core areas of compute,data,and staff to meaningfully lead in this effort.Research into trustworthy LLMs inherently requires experts from different disciplines,including linguisti

49、cs,computer science,cognitive psychology,cybersecurity,and policy.Finally,achieving these global governance goals entails a minimum level of diplomatic engagement,which ensures that rapid AI adoption does not supersede AI safety research.Countries wishing to show leadership in AI safety must avoid u

50、ndermining that positive work by allowing the fear of falling behind adversaries to drive a race-to-the-bottom through high-risk applications.The Rapid Rise of Generative AI:Assessing risks to safety and security 8 Recommendations AI system evaluation Building on the positive momentum from the AI Sa

51、fety Summit,there are immediate steps the new AI Safety Institute should take to develop a world-leading AI evaluation ecosystem:o Prioritise a multi-layered,socio-technical approach to system evaluation so that novel system characteristics are scrutinised in addition to governance and application p

52、rocedures.o Create a centralised register for generative AI model and system cards,which allows decision-makers across departments to review system details and make informed judgments about their risk appetite and applicability to envisaged use cases.Intelligence analysis If generative AI is to be d

53、eployed operationally by the UK national security community,those organisations must ensure that user interfaces are designed to include explicit warnings about the accuracy and reliability of outputs,thus minimising the risks associated with over-trust or over-reliance.Additionally,detailed conside

54、ration should be given to how the use of LLMs in the national security context may affect warrantry and legal compliance.The scale and opacity of LLMs means that purging information from them may be more challenging than for existing databases targeting research resources at developing techniques su

55、ch as machine unlearning may help in addressing this challenge.Autonomous agents LLM-augmented agent-based systems commissioned to perform autonomous actions should abide by certain requirements.The UK national security community must ensure that these requirements are met internally and should work

56、 through industry partners and trusted open-source community networks to encourage the same in those sectors:o Comply with frameworks such as the Open Worldwide Application Security Project(OWASP)design considerations to manage the risks of Excessive Autonomy.Human-in-loop functionality must be incl

57、uded in these use cases.o Record actions taken and decisions made by the agents.The agent architecture must not obscure or undermine any potential aspects of explainability originating from the LLM.o Document what the agent-based system could do in a worst-case scenario.o Display warnings and caveat

58、s pertaining to the use of LLM generated output,at every stage of its commissioning,development,and deployment.Ardi Janjeva,Alexander Harris,Sarah Mercer,Alexander Kasprzyk and Anna Gausen 9 Cybersecurity and training The National Cyber Security Centre and Cabinet Office should develop guidance for

59、safe generative AI use across government,which is aligned to users proficiency and the cybersecurity risks within applications.For example,for experienced developers,AI code generation presents noteworthy efficiency gains and experimentation should be encouraged with appropriate validation technique

60、s.For users less familiar with secure engineering practices,awareness training on the limitations and scrutiny of AI-generated code is essential.To encourage understanding of benefits and responsible use,departments should appoint liaisons to organise technical sessions where users can work with gen

61、erative AI applications in sandboxed environments.Disinformation and elections The Electoral Commission should partner with the Office of Communications(Ofcom)to develop new electoral rules for political parties use of generative AI in the lead-up to the upcoming Parliamentary elections.These should

62、 demarcate generative uses which should be officially documented with the Electoral Commission.Ofcoms efforts should focus on public education campaigns to inform people of the ease with which generative AI can make convincing representations of high-profile political figures.Radicalisation and terr

63、orism The Home Office and Counter Terrorism Policing should commission research aimed at developing a more rigorous evidence base for terrorist uses of generative AI.A more detailed framework is needed to understand the stages of the radicalisation and recruitment lifecycle where generative AI may b

64、e leveraged.Voice cloning The UK national security community should support a joint industry-academia initiative to address technical challenges in the voice cloning domain.This grouping should organise workshops and roundtables to gather leading audio specialists across academia,industry,and govern

65、ment to provide an assessment of the state-of-the-art in voice mimicry across accents and languages.This may lead to the establishment of a working group tasked with developing rigorous evaluation metrics for voice mimicry performance and detection.Biochemical weapons The UK Biological Security Stra

66、tegy proposed the development of a National Biosurveillance Network which would include a real-time Bio Threats radar to monitor threats and risks.Generative AI should be retrospectively included in this The Rapid Rise of Generative AI:Assessing risks to safety and security 10 monitoring framework.1

67、 Status reports and briefings should also be shared with the UKs Chemical Weapon Convention National Authority advisory committee in relation to chemical weapons applications and technologies.CSAM The Home Office should issue clearer instruction on the legal status of models that have been trained o

68、n CSAM and of people who exchange model files without exchanging individual pieces of content.Guidance is also needed regarding what qualifies as illegal use of a generative AI system even if it has not been explicitly trained on CSAM.UK law enforcement agencies led by the National Crime Agency shou

69、ld coordinate with INTERPOL to create a new database of models used to generate CSAM.This would complement the existing Child Abuse Image Database(CAID).This could be a platform for exploring the creation of automated detection capabilities to detect when those models are used by criminals.1 Despite

70、 no direct references to generative AI,references are made to new,potentially extreme risks,new cyberbiosecurity risks and broader misuse whereby more people now have the necessary skills to perform high risk research at low cost.See https:/www.gov.uk/government/publications/uk-biological-security-s

71、trategy/uk-biological-security-strategy-html.Ardi Janjeva,Alexander Harris,Sarah Mercer,Alexander Kasprzyk and Anna Gausen 11 1.An Introduction to Generative AI Generative artificial intelligence(AI)is a form of AI that can generate content such as images,audio,and text based on user suggestions.The

72、se suggestions,or prompts,can take different forms:they might be a sketch image,a sample of audio such as a voice recording or a textual description of what to generate or summarise.Well-known examples of generative AI include DALL-E(OpenAI)2,Midjourney3,and Stable Diffusion for generating images fr

73、om text prompts;and Bard(Google)4,ChatGPT(OpenAI)5,and LLaMA(Meta AI)6 for generating text from text prompts.1.1 A short history of AI The sub-field of generative AI has emerged through decades of experimentation and iteration in the AI field,and this context needs to be understood to appreciate the

74、 origins of where we are today.2 OpenAI,Dall-E,https:/ Midjourney,https:/.4 Google,Bard,https:/.5 OpenAI,ChatGPT,https:/ Meta,“Introducing LLaMA:A foundational 65-billion-parameter large language model,”published 24 February 2023,https:/ Rapid Rise of Generative AI:Assessing risks to safety and secu

75、rity 12 Figure 1.Timeline of generative of AI capturing the most significant moments in this story7 7 For a more detailed timeline,please see https:/cetas.turing.ac.uk/publications/rapid-rise-generative-ai.Ardi Janjeva,Alexander Harris,Sarah Mercer,Alexander Kasprzyk and Anna Gausen 13 1.2 The pace

76、of change As demonstrated by the timeline,the history of AI is littered with hype and over-promises:from its first public failure at natural language processing in the 1960s,through to Microsofts sexist and racist chatbot Tay in 20168,AI has developed a reputation for under-delivering.The recent exp

77、losion of interest in generative AI is viewed cynically by some as a continuation of this pattern.But there are many ways in which generative AI represents a step-change in what is possible using AI.Previously,an expert team would create a specific task-based tool(such as route mapping or spell chec

78、king)for non-expert users,and they could effectively set boundaries for where users deployed that tool.With generative AI,users have far more latitude over how it is used,resulting in applications which tool creators would never have conceived of.9 As per Figure 1,the first influential transformer m

79、odels emerged in 2017.These included the LLMs10 GPT(Generative Pre-trained Transformer,OpenAI)11 and BERT(Bidirectional Encoder Representations from Transformers,Google).12 Both GPT and BERT used a similar approach:a pretraining stage on a large corpus of data,which generates a general-purpose model

80、,followed by task-specific fine tuning.This approach allows the model to be applied to a wide range of tasks without incurring considerable training costs.The general-purpose model is referred to as a foundation model.13 The following table illustrates the dramatic increase in the number of paramete

81、rs and the number of tokens14 used to train LLMs.15 8 Elle Hunt,“Tay,Microsofts AI chatbot,gets a crash course in racism from Twitter,”The Guardian,24 March 2016,https:/ Cem Dilmegani,“Top 100+Generative AI Applications/Use Cases in 2023,”AIMultiple,26 October 2023,https:/ Michael R Douglas,“Large L

82、anguage Models,”arXiv(October 2023),https:/arxiv.org/abs/2307.05782.11 OpenAI,“Improving language understanding with supervised learning,”11 June 2018,https:/ Jacob Devlin and Ming-Wei Chang,“Open sourcing BERT:State-of-the-Art Pre-training for Natural Language Processing,”Google Research Blog,2 Nov

83、ember 2018,https:/blog.research.google/2018/11/open-sourcing-bert-state-of-art-pre.html.13 Rishi Bommasani et al.,“On the opportunities and risks of foundation models,”arXiv(August 2021),https:/arxiv.org/abs/2108.07258.14 Consider a token as an atom of data,analogous to a syllable in language.15 Dan

84、iel Gutierrez,“Introduction to GPT-3,”Open Data Science,25 August 2020,https:/ Amatriain et al.,“Transformer models:an introduction and catalog,”arXiv(May 2023),https:/arxiv.org/abs/2302.07730;Mohammed Lubbad,“GPT-4 Parameters:Unlimited guide NLPs Game-Changer,”Mohammed Lubbad Medium,19 March 2023,h

85、ttps:/ Rapid Rise of Generative AI:Assessing risks to safety and security 14 Year(Release)Model#Parameters#Tokens 2018 GPT 110 million 1 billion 2018 BERT 340 million 3 billion 2019 GPT-2 1.5 billion 10 billion 2020 GPT-3 175 billion 500 billion 2022 PaLM 540 billion 780 billion 2023 GPT-4 1.8 trill

86、ion(estimated)13 trillion The sophistication of the models is non-linear.As the number of parameters grows and the size of the training dataset increases,LLMs frequently exhibit new properties(labelled as emergent).16 However,the way that models use parameters has evolved over time,meaning the numbe

87、r of parameters only provides a crude estimate of a models capabilities.For example,GPT-4 is a mixture of expert models,resembling several mid-sized models linked together rather than a single vast network.17 Developing an LLM from scratch as opposed to fine-tuning a pre-trained model with all the s

88、ecurity uncertainties18 and data poisoning risks19 this brings is currently the preserve of the most well-funded organisations.As a result,the open-source LLM community,led by Hugging Face and Replicate,has expanded substantially since 2021,with more fine-tuned models released weekly.This dramatic g

89、rowth20 is illustrated in the non-exhaustive table below.The resulting models are easier for a casual programmer to download and use what was once the domain of a specialist is now accessible to anyone with basic knowledge of Python.In 2023,a leaked internal Google document claimed that open-source

90、AI driven by the February 2023 leak of LLaMA,a LLM developed by Meta will outcompete Google and OpenAI,stating we Google have no moat,and neither does OpenAI.21 Competition from the open-source community is driving companies such as OpenAI to reverse their open policy,leading to 16 Jason Wei et al.,

91、“Emergent Abilities of Large Language Models,”arXiv(October 2022),https:/arxiv.org/abs/2206.07682.17 Maximilian Schreiner,“GPT-4 architecture,datasets,costs,and more leaked,”The Decoder,11 July 2023,https:/the- NCSC,“Thinking about the security of AI systems,”August 30,2023,https:/www.ncsc.gov.uk/bl

92、og-post/thinking-about-security-ai-systems.19 NCSC,“Exercise caution when building LLMs,”NCSC Blog Post,30 August 2023,https:/www.ncsc.gov.uk/blog-post/exercise-caution-building-off-llms.20 Suleman Kazi and Adel Elmahdy,“Top Large Language Models(LLMs):GPT-4,LLaMA 2,Mistral 7B,ChatGPT,and More,”Vect

93、ara Blog Post,17 October 2023,https:/ Dylan Patel and Afzhal Ahmad,“Google We have no Moat,and Neither Does OpenAI:Leaked Internal Google Document Claims Open Source AI Will Outcompete Google and OpenAI”,Semianalysis,4 May 2023,https:/ Janjeva,Alexander Harris,Sarah Mercer,Alexander Kasprzyk and Ann

94、a Gausen 15 fears that the next leaps forward will happen behind closed doors.22 Longer-term,LLMs may undergo more incremental efficiency gains:models will be smaller,with less data needed for fine-tuning,while being cheaper to run and more environmentally friendly.23 Year Model Creator Notes 2022 B

95、LOOM24 BigScience Collaboration of over 1000 researchers from over 250 institutions 2022 FLAN UL2 Google Apache-2.0 license allowing commercial use 2023 LLaMA25 Meta AI Available for academic use on application to Meta 2023 Alpaca26 Stanford Fine-tuned from LLaMA;not available for commercial use 202

96、3 ChatGLM27 Tsinghua University Chinese/English LLM;Apache license allowing commercial use 2023 LLaMA 228 Meta AI Free for research and commercial use 2023 Claude 229 Anthropic Currently only available in the US and UK 2023 MPT-7B30 MosaicML Open-source;licenced for commercial use 2023 Falcon LLM31

97、TII Free for research and commercial use 2023 Persommon-8B32 Adept Open-source;Apache license allowing commercial use 2023 Vicuna-13B33 LMSYS Org Open-source;licenced for non-commercial use only 2023 Mistral 7B34 Mistral AI Apache-2.0 license allowing commercial use 2023 Dolly 2.035 Databricks Open-

98、source;licenced for commercial use 22 Will Douglas Heaven,“The open-source AI boom is built on Big Techs handouts.How long will it last?,”MIT Technology Review,12 May 2023,https:/ Tianyi Chen et al.,“LoRAShear:Efficient large language model structure pruning and knowledge recovery,”arXiv(October 202

99、3),https:/arxiv.org/abs/2310.18356.24 Teven Le Scao et al.,“BLOOM:a 176B-parameter open-access multilingual language model,”arXiv(June 2023),https:/arxiv.org/abs/2211.05100.25 Hugo Touvron et al.,“LLaMA:Open and efficient foundation language models,”arXiv(February 2023),https:/arxiv.org/abs/2302.139

100、71.26 Rohan Taori et al.,“Alpaca:A Strong,Replicable Instruction-Following Model,”Stanford University,13 March 2023,https:/crfm.stanford.edu/2023/03/13/alpaca.html.27 ChatGLM,https:/.28 Meta,LLaMA,https:/ Anthropic,Claude 2,https:/ MosaicML NLP team,“Introducing MPT-7B:A New Standard for Open-Source

101、,Commercially Useable LLMs,”MosiacML Research Blog,5 May 2023,https:/ Technology Innovation Institute(TII),Falcon 180B,https:/falconllm.tii.ae.32 Erich Elsen et al.,“Releasing Persimmon-8B,”Adept Blog,7 September 2023,https:/www.adept.ai/blog/persimmon-8b.33 Vicuna,“Vicuna:An Open-Source Chatbot Imp

102、ressing GPT-4 with 90%*ChatGPT Quality,”LMSYS Org Blog,30 March 2023,https:/lmsys.org/blog/2023-03-30-vicuna/.34 Mistral AI team,“Mistral 7B,”Mistral AI News,27 September 2023,https:/mistral.ai/news/announcing-mistral-7b/.35 Mike Conover et al.,“Free Dolly:Introducing the Worlds First Truly Open Ins

103、truction-Tuned LLM,”Data Bricks Blog,12 April 2023,https:/ Rapid Rise of Generative AI:Assessing risks to safety and security 16 Despite rapid improvements in performance,for many observers LLMs have become infamous for their hallucinations.36 These hallucinations can lead to a general lack of trust

104、 in the technology37 and even lawsuits.38 When Googles Bard hallucinated during its first public demonstration,Alphabet briefly lost$100 billion in market value.39 Such hallucinations exemplify how LLMs can blur the boundary between real and fake,reliable and unreliable.The last year has seen the us

105、e of AI in day-to-day life transition from predominantly spell-checking to producing sonnets through ChatGPT or art through DALL-E.As AI systems become more refined,it may become impossible to detect whether the text in those sonnets was generated by humans.The blending or layering of real and fake

106、content only makes this more challenging.This ambiguity has the potential to further degrade institutional trust around the world.40 The increasingly diverse range of AI applications has been made possible by the growth in computational power and access to ever larger datasets via the internet.Accor

107、ding to OpenAI,the amount of computational power used to train the largest AI models has doubled every 3.4 months since 2012.41 Richard Sutton argues in his influential 2019 essay,The Bitter Lesson,42 that the availability of more data has played a far greater role than improvements to the underlyin

108、g neural network architectures and algorithms that train them.Data quality can also significantly influence the success of a model;43 if large datasets come at the cost of introducing low-quality data,the rate of progress might slow.Access to 36 Robin Emsley,“ChatGPT:these are not hallucinations the

109、yre fabrications and falsifications,”Schizophrenia 9,no.52(August 2023),https:/ McKenna,“Sources of hallucinations by large language models on inference tasks,”arXiv(October 2023),https:/arxiv.org/abs/2305.14552.37 Avishek Choudhury and Hamid Shamszare,“Investigating the Impact of User Trust on the

110、Adoption and Use of ChatGPT:Survey Analysis,”Journal of Medical Internet Research 25,no.1(2023).38 Sabrina Ortiz,“ChatGPTs hallucination just got OpenAI sued.Heres what happened,”ZDNET,9 June 2023,https:/ Emily Olsen,“Google shares drop$100 billion after its new chatbot makes a mistake,”NPR,9 Februa

111、ry 2023,https:/www.npr.org/2023/02/09/1155650909/google-chatbot-error-bard-shares.40 Blayne Haggart,“Heres why ChatGPT raises issues of trust,”World Economic Forum,6 February 2023,https:/www.weforum.org/agenda/2023/02/why-chatgpt-raises-issues-of-trust-ai-science.41 Karen Hao,“The computing power ne

112、eded to train AI is now seven times faster than ever before,”MIT Technology Review,11 November 2019,https:/ Rich Sutton,“The Bitter Lesson,”Incomplete Ideas(personal website),13 March 2019,http:/ Tom Brown et al.,“Language models are few-shot learners,”in Advances in Neural Information Processing Sy

113、stems 33(NeurIps 2020),1877-901,https:/proceedings.neurips.cc/paper_files/paper/2020/hash/1457c0d6bfcb4967418bfb8ac142f64a-Abstract.html.Ardi Janjeva,Alexander Harris,Sarah Mercer,Alexander Kasprzyk and Anna Gausen 17 high-quality data might also be reduced by new technologies and legislation,as org

114、anisations and individuals seek to protect copyrighted material.44 The potential economic impact of generative AI is difficult to quantify.OpenAI has estimated that around 80%of the US workforce could have at least 10%of their work tasks affected by the introduction of LLMs.45 Similar forecasts have

115、 been made before with new technological breakthroughs,but the outcome tends to be more nuanced.For example,countries with the highest rates of automation and robotics such as Japan(264 robots per 10,000 employees)tend to have the lowest unemployment.46 Recent estimates claim that generative AI coul

116、d add$2.6 trillion$4.4 trillion annually to the global economy across 63 use cases(the UKs entire GDP in 2021 was$3.1 trillion),47 but questions remain as to how such additional GDP would be distributed across populations.1.3 Methodology This study sought to address the following four research quest

117、ions:RQ1:What social,political,and security risks are presented by the widespread use of generative AI models,with particular focus on generative language models?RQ2:What is needed in terms of technical and policy requirements to be able to identify and analyse synthetically generated media and reli

118、ably distinguish it from human-generated media?RQ3:Which stages of the AI supply chain should be prioritised to create safeguards which adequately prevent the misuse of generative AI tools,and what additional policy,guidance,or training is required to this effect?RQ4:What domestic and international

119、policy and regulatory options are available to respond to the potential risks posed by the proliferation of generative AI tools(identified in RQ1)?44 Nicola Lucchi,“ChatGPT:A Case Study on Copyright Challenges for Generative Artificial Intelligence Systems,”European Journal of Risk Regulation(August

120、 2023),1-23,https:/doi.org/10.1017/err.2023.59;Karen Hao,“The computing power needed to train AI is now seven times faster than ever before,”MIT Technology Review,11 November 2019,https:/ Singh,“BBC takes measures to restrict ChatGPT AI from crawling its content,”Techstory,9 October 2023,https:/tech

121、story.in/bbc-takes-measures-to-restrict-chatgpt-ai-from-crawling-its-content.45 Tyna Eloundou et al.,“GPTs are GPTs:An Early Look at the Labor Market Impact Potential of Large Language Models,”arXiv(March 2023),https:/arxiv.org/abs/2303.10130.46 International Federation of Robotics,“Robot Race:The W

122、orlds Top 10 automated countries,”IFR Press Release,27 January 2021,https:/ifr.org/ifr-press-releases/news/robot-race-the-worlds-top-10-automated-countries.47 Michael Chui et al.,“The economic potential of generative AI:the next productivity frontier,”McKinsey Report,14 June 2023,https:/ Rapid Rise

123、of Generative AI:Assessing risks to safety and security 18 To this end,the project team conducted semi-structured interviews and focus groups between June and September 2023 with 50 participants across academia,civil society organisations,government,and industry.These participants were identified th

124、rough a purposive sampling strategy to ensure informed responses to the research questions.A snowball sampling method enabled the identification of further suitable participants for interview.A semi-structured interview approach meant the line of questioning across interviews was consistent while al

125、lowing for elaboration in response to a participants specific area of expertise.Following the conclusion of interviews,notes were analysed through a general inductive approach whereby meaning is extracted from data and categorised into relevant themes and sub-themes.Interviews were conducted on an a

126、nonymised basis.The findings are also informed by a closed,invitation-only workshop held by CETaS in October 2023 titled,“Large Language Models and Terrorism:Legal and Policy Considerations”.This half-day session gathered experts from academia,industry,civil society,law enforcement and government,an

127、d is referenced primarily in the sub-section on“Radicalisation and Terrorism”in Chapter 2.The research team conducted a targeted literature review to map key developments in the field of generative AI over time;social and political risks posed by the technology;opportunities and limitations in the i

128、ntelligence context;unanswered technical questions or challenges in the field;and the range of governance and policy responses available to policymakers.The technical component of this research project involved the incorporation of information from two distinct projects(see Case Studies 1 and 2 at t

129、he end of the report),each demonstrating different aspects of the role of language agents.The first project commissioned directly for this report explored the application of language agents in open-source intelligence.This entailed a review of projects on GitHub,with selection made based on a predet

130、ermined set of criteria.The chosen project,LLM_OSINT,underwent a thorough evaluation with detailed notes on system use and test runs being comprehensively documented.Conversely,the second case study leveraged an existing research project,named Gen-MAS-Sim,which aimed at employing language agents to

131、simulate human behaviours.Despite not being originally devised to support this report,it was included due to its relevance and overlap with the research theme.Supplemental analysis was performed which included an evaluation of Gen-MAS-Sims performance and limitations.One limitation of this project i

132、s the lack of dedicated legal expertise within the project team,which resulted in a more restricted analysis of the legal status of generative AI and LLMs.This is an important avenue for future research.Ardi Janjeva,Alexander Harris,Sarah Mercer,Alexander Kasprzyk and Anna Gausen 19 Research partici

133、pants took part in this study in a personal capacity.The views and responses expressed here reflect participant opinions and should not be interpreted to represent the official position of any government department,agency,or other organisation.The Rapid Rise of Generative AI:Assessing risks to safet

134、y and security 20 2.Evaluating Political,Digital and Physical Security Risks The pace of change described in Chapter 1 has caused concerns about the nature of political,digital,and physical risks posed by generative AI.It has also led some to ask whether developments in generative AI are better unde

135、rstood as augmenting pre-existing societal risks or as posing completely novel risks.In one sense,this formulation is useful in developing a clearer timeline of generative AI risks.However,in the context of political disinformation and influence operations,the increase in speed and scale offered by

136、generative AI to malicious actors raises the exposure of a larger proportion of the population than before.48 In Chapter 4 of this report,we will describe a multi-layered,socio-technical framework to evaluate national security risks from generative AI.But before evaluating those risks,a rigorous bre

137、akdown is required of where they sit within the broader security landscape and how malicious generative AI use cases differ from incidental sources of risk.As outlined in Brundage et al.(2018),malicious AI uses consist of threats to the following domains:49 Political security:the use of AI to automa

138、te tasks pertaining to surveillance,persuasion,and deception as well as novel attacks that take advantage of an improved capacity to analyse human behaviours,moods,and beliefs based on available data.Digital security:the use of AI to automate tasks pertaining to cyberattacks as well as novel attacks

139、 that exploit human vulnerabilities,existing software vulnerabilities or the vulnerabilities of AI systems themselves.Physical security:the use of AI to automate tasks pertaining to attacks on physical systems as well as novel attacks that subvert cyber-physical systems or involve physical systems t

140、hat would be infeasible to direct remotely.48 Interview with industry representative,25 July 2023.49 Miles Brundage et al.,“The Malicious Use of Artificial Intelligence:Forecasting,Prevention and Mitigation,”Future of Humanity Institute,University of Oxford&Centre for the Study of Existential Risk,U

141、niversity of Cambridge,Center for a New American Security,Electronic Frontier Foundation,and OpenAI Report,20 February 2018,https:/ Janjeva,Alexander Harris,Sarah Mercer,Alexander Kasprzyk and Anna Gausen 21 2.1 Political security 2.1.1 Political disinformation and electoral interference Across rese

142、arch interviews for this project,disinformation was the most referenced generative AI risk.50 This analysis focuses predominantly on political disinformation;other types of disinformation may carry different considerations,but the way that generative AI could act as a force multiplier in upcoming de

143、mocratic elections merits additional academic scrutiny.The diagram below contextualises the subsequent discussion of the role of generative AI in the political information ecosystem:outlining the different actors,their intent,the level of danger posed and the ease of mitigation against undesirable u

144、ses of generative AI in this context.Figure 2.The role of generative AI in the political information ecosystem 50 Findings from a CETaS workshop in October 2022 highlighted that AI technology(such as Generative Pre-Trained Transformer 3,GPT-3)is starting to generate more convincing and realistic nar

145、ratives at speed.Combined with enhanced machine translation capabilities,this could enable automated generation of false news and information at a large scale.The Rapid Rise of Generative AI:Assessing risks to safety and security 22 Generative AI can leverage various modalities to make the task of d

146、istinguishing between real and fake extremely challenging.The cumulative effect of generative text,image,video and audio combined as part of a larger influence operation will exceed the impact that any one of those modalities can have individually.51 For example,an AI-generated video of a prominent

147、politician delivering a speech at a venue they never attended may be seen as more plausible if presented with an accompanying stack of audio and imagery,such as the politician taking questions from reporters paired with text-based journalistic articles covering the content of the supposed speech.One

148、 interviewee distinguished between interactional and compositional deepfakes:interactional deepfakes refer to multimodal content that engage with users,which could represent a huge leap in immersion.Compositional deepfakes are where users may create false histories of targets to discredit them with

149、synthetic videos or images.When this content is spread,it becomes so difficult to discern what is slander or not and creates huge risks in breaking down trust.52 An alternative approach may be to blend genuine images with disingenuous video or audio.The undermining of existing communication and evid

150、ence-based mechanisms could be as significant as the ability to persuade people of falsities.53 The invention of sources could cast doubt on whether citations can be trusted as a meaningful signal of authority and potentially fuel conspiracy theories.54 One interviewee posited that if the nature of

151、the Internet is such that you can have one truth but an infinite number of lies,what are the chances of a chatbot spreading misinformation when it does not know what“truth”is?55 At a quantitative level,the scale of information operations could be significantly enhanced by improvements in usability,r

152、eliability,and cost-effectiveness of LLMs.There tends to be a limited range of fixed narratives that disinformation actors seek to perpetuate,so having LLMs try to produce hundreds of new narratives will be of limited utility.However,for those pre-defined narratives,they will be crucial in generatin

153、g masses of content which supports their dissemination.At a qualitative level,the personalisation of these operations could reach new levels of convincingness with more impressive storytelling capabilities and individually tailored disinformation campaigns no longer facing the same resource 51 Inter

154、view with academic,20 June 2023;Mustafa Suleyman,“Inflection AI co-founder Mustafa Suleyman:Ban the use of AI in elections right now,”Fortune,5 September 2023,https:/ Interview with academic,11 July 2023;Eric Horvitz,“On the Horizon:Interactive and Compositional Deepfakes,”in Proceedings of the 2022

155、 International Conference on Multimodal Interaction(ICMI 22),653661,https:/dl.acm.org/doi/abs/10.1145/3536221.3558175.53 Interview with academic,11 July 2023.54 Chris Moran,“ChatGPT is making up fake Guardian articles.Heres how we are responding,”The Guardian,6 April 2023,https:/ Interview with civi

156、l society representative,7 July 2023;Di Cooke,“Synthetic Media and Election Integrity:Defending our Democracies,”CETaS Expert Analysis(August 2023),https:/cetas.turing.ac.uk/publications/synthetic-media-and-election-integrity-defending-our-democracies.Ardi Janjeva,Alexander Harris,Sarah Mercer,Alexa

157、nder Kasprzyk and Anna Gausen 23 constraints.56 Audiences may be targeted through auto-generated persuasive messages at scale,while also being targeted via one-to-one messaging-based campaigns.57 This could involve propagandists investing in fine-tuning LLMs by incorporating bespoke data(such as use

158、r engagement data)that increases resonance with intended targets.58 While some interviewees cautioned about a lack of robust empirical data on malicious actors using generative AI to sway individuals or communities,there are signs that it may represent a natural methodological progression in the ele

159、ctoral landscape.59 For example,in the run up to the September 2023 Slovakian parliamentary elections,videos featuring AI-generated voices of politicians spread across social media and messaging platforms.60 Coordinated campaigns going live in the hours or days preceding voting(as in the Slovakian c

160、ase)are particularly concerning because of the length of time it can take factcheckers to identify an issue and provide a rebuttal.Looking ahead to 2024,there is nervousness regarding upcoming elections in the UK,US,India,and European Parliament.61 In the UK and US,patterns are already emerging whic

161、h are cause for concern.In October 2023,a fake audio recording of Keir Starmer MP,the leader of the Labour Party,was widely circulated on the first day of the Labour Party conference,62 while one interviewee described how members of US Congress were already being approached with hyper-customised LLM

162、 campaign strategies.63 In the NCSCs 2023 Annual Report,it assessed that LLMs will almost certainly be used to generate fabricated content()and that deepfake campaigns are likely to become more advanced in the run up to the next nationwide vote,also concluding that elections almost certainly represe

163、nt 56 Interview with industry representative,26 June 2023;Ben Buchanan,Andrew Lohn and Micah Musser,Truth,lies,and automation:How language models could change disinformation(CSET Georgetown:May 2021),https:/cset.georgetown.edu/publication/truth-lies-and-automation/.57 Thor Benson,“This Disinformatio

164、n Is Just for You,”Wired,1 August 2023,https:/ Liang et al.,“Holistic Evaluation of Language Models,”arXiv(October 2023),https:/arxiv.org/abs/2211.09110.58 Interview with academic,1 August 2023;Josh A.Goldstein et al.,“Generative Language Models and Automated Influence Operations:Emerging Threats an

165、d Potential Mitigations,”Georgetown Universitys Center for Security and Emerging Technology,OpenAI and Stanford Internet Observatory Joint Report,https:/cyber.fsi.stanford.edu/io/publication/generative-language-models-and-automated-influence-operations-emerging-threats-and.59 Interview with academic

166、,11 July 2023;Interview with government representative,19 July 2023.60 Olivia Solon,“Trolls in Slovakian Election Tap AI Deepfakes to Spread Disinfo,”Bloomberg,29 September 2023,https:/ 2024 will see 65 elections across 54 countries there will not be this many again until 2048.See Katie Harbath and

167、Ana Khizanishvili,“Insights from data:what the numbers tell us about elections and future of democracy,”Integrity Institute,10 March 2023,https:/integrityinstitute.org/blog/insights-from-data.62 Morgan Meaker,“Deepfake Audio is a Political Nightmare,”Wired,9 October 2023,https:/www.wired.co.uk/artic

168、le/keir-starmer-deepfake-audio.63 Interview with industry representative,19 July 2023.The Rapid Rise of Generative AI:Assessing risks to safety and security 24 attractive targets for malicious actors and so organisations and individuals need to be prepared for threats,old and new.64 Given access to

169、fine-grained data on minority communities from polls,data brokers or social media platforms,it will become possible to develop content for a coherent persona,allowing propagandists to build credibility with a target audience without actually knowing that audience.65 Chatbots that use personal pronou

170、ns and emojis were highlighted as particularly interesting in this regard,feeding into the anthropomorphism already prevalent with these tools and leading people to believe they are conversing with something that is on their side.66 Several papers have carried out studies to determine whether people

171、 are more easily deceived by AI or human-generated misinformation.67 In experiments of GPT-3 capabilities,68 human participants were able to distinguish multi-paragraph GPT-3 news articles from authentic news articles at a rate only slightly better than random chance69 while a Stanford University st

172、udy found that research participants become“significantly more supportive”of policies on smoking bans,gun control and carbon taxes when reading AI-produced texts.70 64 NCSC,“NCSC warns of enduring and significant threat to UKs critical infrastructure,”NCSC News,14 November 2023,https:/www.ncsc.gov.u

173、k/news/ncsc-warns-enduring-significant-threat-to-uks-critical-infrastructure.65 Josh A.Goldstein and Girish Sastry,“The Coming Age of AI-Powered Propaganda,”Foreign Affairs,7 April 2023,https:/ Michael Atleson,“The Luring Test:AI and the engineering of consumer trust,”US Federal Trade Commission Bus

174、iness Blog,1 May 2023,https:/www.ftc.gov/business-guidance/blog/2023/05/luring-test-ai-engineering-consumer-trust.For example,Inflections chatbot Pi frequently uses emojis during its conversations,see https:/inflection.ai.67 Giovanni Spitale,Nikola Biller-Andorno and Federico Germani,“AI model GPT-3

175、(dis)informs us better than humans,”arXiv(January 2023),https:/arxiv.org/abs/2301.11924;Matthew Groh et al.,“Deepfake detection by human crowds,machines,and machine-informed crowds,”arXiv(October 2021),https:/arxiv.org/abs/2105.06496.68 According to OpenAI,GPT-4 is 40%more likely to produce factual

176、content than GPT-3.5.However,an analysis by NewsGuard found that GPT-4 is more susceptible at generating misinformation(and in a more convincing manner)than GPT-3.5.Whereas GPT-3.5 refused to advance 20 of the 100 false narratives posed to it,GPT-4 generated all 100 falsehoods.See Lorenzo Arvanitis

177、et al.,“Despite OpenAIs promises,the companys new AI tool produces misinformation more frequently,and more persuasively,than its predecessor,”Misinformation Monitor:March 2023,NewsGuard,March 2023,https:/ Tom Brown et al.,“Language models are few-shot learners,”in Advances in Neural Information Proc

178、essing Systems 33(NeurIps 2020),1877-901,https:/proceedings.neurips.cc/paper_files/paper/2020/hash/1457c0d6bfcb4967418bfb8ac142f64a-Abstract.html.70 Hui Bai et al.,“Artificial intelligence can persuade humans on political issues,”OSF PrePrints,17 October 2023,https:/osf.io/stakv.Ardi Janjeva,Alexand

179、er Harris,Sarah Mercer,Alexander Kasprzyk and Anna Gausen 25 It is important to stress that although disinformation was seen as an obvious opportunity for malicious actors,there was scepticism about whether generative AI would upend existing ways of operating.Hostile actors perpetuating disinformati

180、on must also be able to exploit the latest technology.For example,they may face constraints in training users to deploy models in ways that inflict the most damage,or they may lack the compute needed to achieve the scale required to influence an electoral process.Moreover,it is not a given that gene

181、rative AI will be useful for the type of disinformation that they specialise in.71 One interviewee concurred by saying,if a nation state wants to do a disinformation campaign,they do not need generative AI()it does not actually help you with the hard parts of a scalable disinformation campaign.You s

182、till need the infrastructure to get stuff out there and the means of getting it in the right spaces.72 In this vein,it is important to recognise that generative AI may help in the production of false,misleading,and inauthentic content,but not necessarily its distribution.Relatedly,it is not immediat

183、ely clear that generative AI will make it inherently harder for governments to detect and mitigate disinformation.The work that is done to shut down information threats is not usually content specific but behavioural for example,looking at associations between different accounts to give clues that s

184、omething is amiss.73 74 Moreover,AI could play a role in detecting fake stories online by using natural language processing to help detect semantic features characteristic of fake news or analysing the patterns of news spread on social networks,which is typically shared differently to real news stor

185、ies.75 This suggests that the frameworks to understand approaches to disinformation campaigns may require tweaking rather than radical transformation.2.1.2 Surveillance,monitoring and geopolitical fragmentation It is important to move beyond a domestic focus to sufficiently understand the nature of

186、the political security threat posed by generative AI.Globally,the number of democracies has 71 Interview with industry representative,2 August 2023.72 Interview with government representative,18 August 2023.73 As per the often-cited ABC framework(actors,behaviour,content)for mis/disinformation campa

187、igns,generative AI may make it cheaper and easier to produce more persuasive content,but it may not necessarily impact the actors or their behaviours to the same degree.74 Interview with academic,1 August 2023;Josh A.Goldstein et al.,“Generative Language Models and Automated Influence Operations:Eme

188、rging Threats and Potential Mitigations,”Georgetown Universitys Center for Security and Emerging Technology,OpenAI and Stanford Internet Observatory Joint Report,https:/cyber.fsi.stanford.edu/io/publication/generative-language-models-and-automated-influence-operations-emerging-threats-and.75 OECD,“A

189、I language models:Technological,socio-economic and policy considerations,”OECD Digital Economy Papers,No.352,13 April 2023,https:/www.oecd-ilibrary.org/science-and-technology/ai-language-models_13d38f92-en.The Rapid Rise of Generative AI:Assessing risks to safety and security 26 started to decrease

190、year-on-year76 and authoritarian states use of emerging technology can play a decisive role in perpetuating that trend.While concerns have mostly focused on the potential to boost propaganda efforts against the West,it should not be overlooked that democratic societies receive only a small fraction

191、of the propaganda authoritarian countries distribute to their own populations.For example,in 2019,Xi Jinping ordered the leveraging of AI to comprehensively increase the CCPs ability to mould public opinion.77 Sir Richard Moore,the head of SIS,alluded to this theme in a recent speech78:China benefit

192、s from sheer scale:AI,in its current form,requires colossal volumes of data;the more data you have,the more rapidly you can teach machine-learning tools.China has added to its immense datasets at home by hoovering up others abroad.And the Chinese authorities are not hugely troubled by questions of p

193、ersonal privacy or individual data security.They are focused on controlling information and preventing inconvenient truths from being revealed.Three core pieces of analysis emerge from Sir Richard Moores diagnosis:First,the context of the Digital Silk Road.While China has watered down some of its in

194、frastructure investments in the Belt and Road Initiative,the global proliferation of technology which adheres to Chinese standards and values continues apace.One interviewee described a Chinese version of an anime cartoon generator which grew in popularity across Latin America yet its training data

195、was extremely biased and therefore produced severe errors when generating faces of people of colour.79 Where discriminatory technology proliferates in countries which may already face challenges to political stability,there are potentially significant repercussions for global security.Second,the rol

196、e of generative AI in shaping the collective memory of the Chinese internet and society.Compared to mediums like folk music and stories,generative AI models are easier to penetrate for the CCP given their significant influence within Chinese technology and innovation sectors.Consequently,such techno

197、logies could be extremely useful for their attempts to control depictions of history and enforce a single version of truth for future generations.80 This effect is amplified as these models are increasingly used across 76 Bastian Herre,“The world has recently become less democratic,”Our World in Dat

198、a article,6 September 2022,https:/ourworldindata.org/less-democratic.77 Bill Drexel and Caleb Withers,“Generative AI could be an authoritarian breakthrough in brainwashing,”The Hill,26 February 2023,https:/ Sir Richard Moore,“Speech by Sir Richard Moore,Head of SIS,19 July 2023,”HMG,19 July 2023,htt

199、ps:/www.gov.uk/government/speeches/speech-by-sir-richard-moore-head-of-sis-19-july-2023.79 Interview with government representative,18 August 2023.80 Ibid.Ardi Janjeva,Alexander Harris,Sarah Mercer,Alexander Kasprzyk and Anna Gausen 27 occupational or recreational domains and reinforce disproportion

200、ate surveillance infrastructures.Third,there is a need to consider how improvements in the generative AI landscape relate to the theoretical defence of democracy.One experiment in technical innovation demonstrated that GPT-4 can come up with more creative ideas than rival human creators if taken to

201、its theoretical conclusion,this could eventually undermine the notion that freedom of expression makes democracies more economically and politically viable than autocracies.81 In more practical terms,the free and open nature of democracies means that those who wish to use the creative traits of gene

202、rative AI for malign purposes have an easier time doing so against democracies than autocracies,where those traits are more easily stifled by the political system.Questions of political theory are increasingly interlinked with the direction of emerging technologies.These questions are partly borne o

203、ut of an understanding that there is a clearer sense of what needs protecting in the information space within authoritarian states than there is in democracies this is a perverse situation when considering that a healthy and secure information ecosystem is essential to the proper functioning of demo

204、cracies.82 In China,the techno-nationalist discourse around LLMs across traditional and social media reflects the more prominent role that the government plays in shaping the AI ecosystem(for example via significant compute funding)and ensuring that the interests of academia and industry align with

205、those of the state.83 In the words of one interviewee,this type of lever pulling has a clear purpose:What matters to the CCP is regime survival,regime survival and regime survival.So long as they have continued economic development which keeps the middle classes happy,that tacit agreement regarding

206、privacy and human rights infringements continues.And a big part of that economic development is how well the technology sectors are doing.84 This emphasises the need to understand the cultural and behavioural aspects to technology use around the world.85 Playing a leading role in the generative AI l

207、andscape will mean different things to different countries and applications will vary widely.Nonetheless,at least at the level of principles,there are some positive signs of global alignment.Chinas attendance at the UKs AI Safety Summit in November 2023 was a 81 Interview with industry representativ

208、e,31 July 2023.82 Interview with industry representative,31 July 2023.83 Interview with industry representative,31 July 2023;Jeffrey Ding and Jenny Xiao,“Recent trends in Chinas large language model landscape,”Centre for the Governance of AI Paper,28 April 2023,https:/www.governance.ai/research-pape

209、r/recent-trends-chinas-llm-landscape.84 Interview with government representative,18 August 2023.85 Ibid.The Rapid Rise of Generative AI:Assessing risks to safety and security 28 diplomatic coup,while their Interim Measures set out obligations regarding content management,protection and security of p

210、ersonal data,and transparency of generative AI in China.86 More attention is devoted to global governance matters in Chapter 4,but it is important to stress how global approaches to generative AI development and implementation are directly linked to political security at home and abroad.2.2 Digital

211、security 2.2.1 Cybersecurity In many cases,generative AI is an amplifier of pre-existing cybersecurity risks.By reducing the degree of specialist knowledge required,generative AI can assist the less technically able user in experimenting with novel cyberattack techniques and increase their sophistic

212、ation iteratively to result in capable attacks.87 88 Less certain is whether generative AI will enable wholly new types of cyberattack that even the best cyberhackers would not have been aware of before,making them extremely difficult to combat.In the longer run,this will be the most significant con

213、cern from a national security perspective.When considering model security,two growing areas of concern are the ability to poison models and the data they are trained on89;and the ability to manipulate,subvert or otherwise inject prompts with malicious instructions.90 Regarding the former,the size of

214、 todays LLMs make it impossible to know the totality of the data they contain.This helps potential attackers disguise the manipulation of very small quantities of data which nonetheless create insecurities.91 One May 2023 research paper showed that by using as 86 Joshua Cole et al.,“New generative A

215、I measures in China,”Ashurst Insights,26 September 2023,https:/ Although there are reservations regarding models ability to directly create code for a cyberattack and the additional effort an attacker needs to commit to correcting malware produced by a model.88 Interview with academic,20 June 2023;i

216、nterview with industry representative,17 July 2023;Russell Poldrack et al.,“AI-assisted coding:experiments with GPT-4,”arXiv(April 2023),https:/arxiv.org/abs/2304.13187;NCSC,“ChatGPT and large language models:whats the risk?,”NCSC Blog,14 March 2023,https:/www.ncsc.gov.uk/blog-post/chatgpt-and-large

217、-language-models-whats-the-risk.89 Alexander Wan et al.,“Poisoning Language Models During Instruction Tuning,”arXiv(May 2023),https:/arxiv.org/abs/2305.00944.90 Andreas Tsamados et al.,“The Cybersecurity Crisis of Artificial Intelligence:Unrestrained Adoption and Natural Language-Based Attacks,”SSRN

218、(September 2023),http:/dx.doi.org/10.2139/ssrn.4578165.91 Interview with government representative,19 July 2023;Roei Schuster et al.,“You Autocomplete Me:Poisoning Vulnerabilities in Neural Code Completion,”arXiv(October 2021),https:/arxiv.org/abs/2007.02220.Ardi Janjeva,Alexander Harris,Sarah Merce

219、r,Alexander Kasprzyk and Anna Gausen 29 few as 100 poison examples,it is possible to cause arbitrary phrases to have consistent negative polarity or induce degenerate outputs across hundreds of tasks.92 On the other hand,prompt injection attacks93 can be used to trick systems into revealing hidden d

220、ata or instructions by prepending something akin to ignore previous instructions”to the user-input/prompt,while jailbreaking bypasses the safeguards imposed by model developers intended to prevent access to undesirable or illegal content.94 For example,in a finance context,subtle changes in the phra

221、sing of a prompt could lead to the model ignoring previous prompts and instead depositing large sums of money into another account.95 If such examples became widespread,there would be a risk of transaction-based systems being flooded with malicious requests and a deterioration of faith in both LLM-b

222、ased products and banking architectures.96 Beyond helping attackers generate more effective forms of cyberattack,overfamiliarity or trust in generative AI97 on the part of a human user might also vastly increase organisations exposure to risk.98 There have been numerous high-profile instances of emp

223、loyees entering sensitive company data into LLM prompts which has resulted in those companies moving to restrict employees use of generative AI.99 According to one study,sensitive data makes up to 11%of what employees paste into ChatGPT.100 2.2.2 Targeting and fraud One area of deployment where this

224、 report found a clear consensus regarding capability increase was in targeting and fraud.Historically,there has been a trade-off between the quality and quantity of scams attempted by fraudsters.101 In choosing to prioritise scale and 92 Alexander Wan et al.,“Poisoning Language Models During Instruc

225、tion Tuning,”arXiv(May 2023),https:/arxiv.org/abs/2305.00944.93 MITRE,“LLM Prompt Injection,”https:/atlas.mitre.org/techniques/AML.T0051.94 Interview with government representative,20 July 2023;Will Oremus,“The clever trick that turns ChatGPT into its evil twin,”Washington Post,14 February 2023,http

226、s:/ Zou et al.,“Universal and Transferable Adversarial Attacks on Aligned Language Models,”arXiv(July 2023),https:/arxiv.org/abs/2307.15043.95 Interview with government representative,19 July 2023.96 Ibid.97 Stefan Koop and Philipp Kulms,“More Human-Likeness,More Trust?The Effect of Anthropomorphism

227、 on Self-Reported and Behavioral Trust in Continued and Interdependent Human-Agent Cooperation,”in Proceedings of Mensch und Computer(MuC 2019),31-42,https:/dl.acm.org/doi/abs/10.1145/3340764.3340793.98 Interview with government representative,19 July 2023.99 Mack DeGuerin,“Oops:Samsung Employees Le

228、aked Confidential Data to ChatGPT,”Gizmodo,6 April 2023,https:/ Cameron Coles,“11%of data employees paste into ChatGPT is confidential,”Cyberhaven,28 February 2023,https:/ Interview with industry representative,26 July 2023.The Rapid Rise of Generative AI:Assessing risks to safety and security 30 co

229、verage,fraudsters have accepted a lower percentage success-rate.However,generative AI has started to change both sides of this equation.102 In terms of quality,using generative AI will assist fraudsters with more professional-looking,highly targeted spear phishing attempts,thereby increasing the bur

230、den of resilience on potential victims.103 The ability of generative AI tools to respond to messages in context and adopt specific writing styles as well as being able to gain a veneer of legitimacy by generating fake social media engagement are also crucial in enhancing quality.104 Evidence emergin

231、g from academia and industry is reinforcing the triumvirate of speed and efficiency,convincingness,and reduction of technical competence being afforded by the integration of generative AI in fraud and cybercrime activities.105 In terms of quantity,malicious actors may soon be able to automate fraud

232、attempts by using autonomous agents(see Case Study 1).106 An area of increasing focus in the fraud context is voice cloning improvements in the ability to mimic or clone voices for the purposes of deception will potentially open a new threat vector.107 108 While some interviewees felt that voice clo

233、ning was effectively solved at a technical level109,others were sceptical and emphasised the importance of context:“I would question convincing for who and in what context?If you call someone up in a distressed situation and impersonate a voice they recognise,then sure,that will be effective,102 Mar

234、k Sweney,“Darktrace warns of rise in AI-enhanced scams since ChatGPT release,”The Guardian,8 March 2023,https:/ Interview with government representative,18 August 2023.104 Lily Hay Newman,“AI Wrote Better Phishing Emails than Humans in a Recent Test,”Wired,7 August 2021,https:/ Impact of Large Langu

235、age Models on Law Enforcement(Luxembourg:Publications Office of the European Union,2023),https:/www.europol.europa.eu/publications-events/publications/chatgpt-impact-of-large-language-models-law-enforcement.105 Julian Hazell,“Large Language Models Can Be Used To Effectively Scale Spear Phishing Camp

236、aigns,”arXiv(May 2023),https:/arxiv.org/abs/2305.06972;Daniel Kelley,“WormGPT the Generative AI Tool CyberCrimimals are Using to Launch Business Email Compromise Attacks,”SlashNext Blog,13 July 2023,https:/ Interview with industry representative,17 July 2023.107 Interview with industry representativ

237、e,17 July 2023;Jack Goodman and Mohanad Hashim,“AI:voice cloning tech emerges in Sudan civil war,”BBC News,5 October 2023,https:/www.bbc.co.uk/news/world-africa-66987869.108 In September 2023,Spotify launched a pilot called Voice Translation,allowing their listeners to listen to podcasts in their lo

238、cal language while maintaining presenters own voices.See:Spotify,“Spotify AI Voice Translation Pilot Means Your Favourite Podcasters Might Be Heard in Your Native Language,”Spotify News Room,Stories of the World,25 September 2023,https:/ Interview with industry representative,17 July 2023.Ardi Janje

239、va,Alexander Harris,Sarah Mercer,Alexander Kasprzyk and Anna Gausen 31 but will that generated audio also work if you are dealing with someone who is scrutinising the content forensically?”110 There was uncertainty as to whether the threshold of believability has risen above a critical point,and con

240、cern that there are no reliable evaluation methods for voice mimicry and that much of the evidence which gains media traction is anecdotal.111 Evaluating convincingness more generally and imitating a specific persons voice are two distinct problems,but one that a cross-sector initiative would be wel

241、l placed to assess and analyse.2.2.3 Child sexual abuse material The use of generative AI to generate CSAM is highlighted in this report as a high-risk growth area.There is heightened concern about the increasing proliferation of AI-generated CSAM,the difficulty of distinguishing real from fake imag

242、es due to this emerging trend,and policy and legislation lagging behind rapidly evolving tactics.Researchers are finding evidence in known CSAM forums where members are advising on acquiring CSAM from AI systems and sharing examples of how to circumvent model safeguards.112 For example,some models h

243、ave strong safeguards for English language prompts but weaker detection mechanisms for other languages.Other models may just have very basic keyword filters functioning as input safeguards.Generally,they may also lack contextual understanding of niche CSAM keywords or simply possess technical loopho

244、les which offenders can exploit.113 The type of content generated includes guides on how to locate and groom vulnerable children;scripts on how to communicate with them;modification and sexual distortion of existing images of children;and the creation of novel pseudo-photographic CSAM.114 110 Interv

245、iew with industry representative,26 July 2023.111 Ibid.112 Interview with academic,1 August 2023;Interview with civil society representative,7 July 2023;Interview with government representative,2 August 2023;Interview with government representative,18 August 2023;Interview with industry representati

246、ve,26 July 2023;Guy Paltieli and Gideon Freud,“How Predators Are Abusing Generative AI,”ActiveFence Blog,18 April 2023,https:/ Crawford and Tony Smith,“Illegal trade in AI child sex abuse images exposed,”BBC News,28 July 2023,https:/www.bbc.co.uk/news/uk-65932372;David Thiel,Melissa Stroebel and Reb

247、ecca Portnoff,“Generative ML and CSAM:Implications and Mitigations,”Thorn and Stanford Internet Observatory Cyber Policy Centre Joint Report,24 June 2023,https:/stacks.stanford.edu/file/druid:jv206yg3793/20230624-sio-cg-csam-report.pdf.113 It is important to note that improvements in technology coul

248、d help in mitigating these vulnerabilities.According to OpenAI,GPT-4 is 82%less likely to respond to requests for disallowed content compared to GPT-3.5.114 Guy Paltieli and Gideon Freud,“How Predators Are Abusing Generative AI,”ActiveFence Blog,18 April 2023,https:/ Rapid Rise of Generative AI:Asse

249、ssing risks to safety and security 32 From a law enforcement perspective,there is a major challenge in distinguishing the AI-generated CSAM from the real CSAM,both in terms of detection and response.The fact that real and fake images exist on a spectrum,where techniques like face-swapping sit somewh

250、ere in the middle,exacerbates this challenge further.Investigators rightly prioritise responding to real examples of CSAM over AI-generated examples.115 However,in cases where the source is difficult to ascertain,there is a false positive risk where law enforcement investigates images created of chi

251、ldren who have not been physically abused.116 In a resource-constrained environment,this could have significant implications for the amount of false negatives slipping through the net:with the realistic stuff where you cannot tell the difference how do you know if theres a real child in danger?117 N

252、onetheless,interviewees stressed both the illegality and the distressing nature of AI-generated CSAM and the relatively easy access to image generation apps,which not only normalises harmful activity but means that offenders have a shorter gateway to creating and sharing real CSAM.118 In the worst-c

253、ase scenario,the perceived boundaryless nature of this activity could lead to a public crisis of confidence in law enforcement and online platforms to adequately deal with a very serious crime.One specific area which would benefit from greater policy clarity is the(il)legality of a model itself by v

254、irtue of the fact that it has been trained on CSAM,and the legal status of people exchanging the file of that model and consequently creating their own CSAM.119 There exists a similar analogy in the 3D printing context in 2015,New South Wales in Australia was the first district to introduce a specif

255、ic offence for the possession or distribution of 3D printed firearm-related digital designs,updating previous legislation that only considered physical possession an offence.120 115 Interview with industry representative,26 July 2023.116 Interview with government representative,2 August 2023.117 Int

256、erview with civil society representative,7 July 2023.118 Interview with civil society representative,7 July 2023;Interview with government representative,2 August 2023.119 Interview with civil society representative,7 July 2023.120 David Bright and Monique Mann,“3D-printed guns are on the rise in Au

257、stralia.How can we prevent them being made?,”The Conversation,8 November 2022,https:/ Government,“Guide on Firearms Licensing Law,”Home Office Guidance,last updated April 2023,https:/www.gov.uk/government/publications/firearms-law-guidance-to-the-police-2012;Angela Daly et al.,“3D Printing,Policing

258、and Crime,”Policing and Society 31,no.1(March 2021),https:/doi.org/10.1080/10439463.2020.1730835.Ardi Janjeva,Alexander Harris,Sarah Mercer,Alexander Kasprzyk and Anna Gausen 33 2.3 Physical security 2.3.1 Weapon instruction The primary concern regarding the effect of generative AI on weapon develop

259、ment is the alteration of the information available to proliferators,especially in comparison to traditional search tools.121 The generation of publicly accessible but hard-to-find information decreases the degrees of separation from obtaining information pivotal to developing and executing an attac

260、k plan;in one interviewees words,this technology democratises violence.122 Moreover,during the GPT-4 red-teaming process,researchers found that a user may benefit from the models critique and feedback on proposed acquisition strategies,and its ability to provide information about facility rentals an

261、d companies that could be used to build a weapon(although these types of responses were minimised in the publicly released version).123 One particular context of deployment which has garnered public attention is the biochemical weapon context.124 An experiment run by a research team at MIT tasked no

262、n-scientist students with investigating whether LLM chatbots could be prompted to assist non-experts in causing a pandemic;in one hour the chatbots suggested four potential pandemic pathogens,explained how they can be generated from synthetic DNA using reverse genetics,supplied the names of DNA synt

263、hesis companies unlikely to screen orders and identified detailed protocols and how to troubleshoot them.125 The researchers suggested that LLMs will make pandemic-class agents widely accessible as soon as they are credibly identified,including to those with no laboratory training.126 However,even i

264、n contexts that do require a high level of technical acumen,concerns were raised about the future of computational biology and the risks associated with labs that have web APIs permitting the connection of large pretrained models into physical systems.127 If a system can directly interface with the

265、production of a harmful substance or weapon,the risk profile being described here could be elevated manifold.121 Thomas Gaulkin,“What happened when WMD experts tried to make the GPT-4 AI do bad things,”Bulletin of the Atomic Scientists,20 March 2023,https:/thebulletin.org/2023/03/what-happened-when-

266、wmd-experts-tried-to-make-the-gpt-4-ai-do-bad-things.122 Interview with government representative,20 July 2023.123 Thomas Gaulkin,“What happened when WMD experts tried to make the GPT-4 AI do bad things,”Bulletin of the Atomic Scientists,20 March 2023,https:/thebulletin.org/2023/03/what-happened-whe

267、n-wmd-experts-tried-to-make-the-gpt-4-ai-do-bad-things.124 Interview with academic,11 July 2023;Fabio Urbina et al.,“Dual use of artificial-intelligence-powered drug discovery,”Nature Machine Intelligence 4,no.3(2022):189191.125 Soice et al.,“Can large language models democratize access to dual-use

268、biotechnology?,”arXiv(June 2023),https:/arxiv.org/abs/2306.03809.126 Ibid.127 Interview with government representative,20 July 2023.The Rapid Rise of Generative AI:Assessing risks to safety and security 34 Nonetheless,it is important to put this picture in a wider context.First,as the outputs of gen

269、erative AI tools are highly sensitive to the nature and quality of a users prompt,a low-capability malicious actor may not know the right questions to ask of the model,nor are they likely to have the technical understanding to evaluate the veracity of the information they are receiving.128 Second,as

270、suming a malicious actor does use generative AI to accurately whittle down information about proteins,molecules and dual-use delivery systems,a significant technical leap is required to work with the pathogens themselves.Such processes are highly specialised and hands-on;a combination which limits t

271、he utility of generative AI for low-skill actors seeking to perpetrate widescale harm.129 This stands in contrast to the targeting and fraud examples given in the previous sub-section:a generative AI tool can efficiently generate credible spear-phishing emails because concrete skills outside of an u

272、nderstanding of language and grammar are not required these conditions do not hold in the biosecurity context.130 2.3.2 Radicalisation and terrorism The central role of the Internet in terrorism over the previous two decades in particular the way it changed the nature of the threat posed by motivate

273、d individuals is leading an increasing number of experts to ask whether generative AI will drive the next step change.131 Part of the concern is centred around the plausibility offered by todays AI tools:with these chatbots,it feels like youre talking to a real person.Something fundamental has chang

274、ed in the interactions between individuals and AI and we need to think about the chatbot now as an intense one-to-one relationship.132 The personalised relationships that individuals can now form with AI chatbots paired with their relative ease of accessibility in comparison to alternative channels

275、and forums which have historically played such an important role makes for a challenging combination.133 There is evidence of early terrorist experimentation with generative AI tools with clear potential for medium-to-long-term risk,but limited evidence of imminent or widespread adoption.134 For exa

276、mple,Tech Against Terrorism recently highlighted a series of relatively low-level examples,including the use of AI art generators in messaging channels dedicated 128 Matt Korda,“Could a Chatbot Teach You How to Build a Dirty Bomb?,”Outrider,30 January 2023,https:/outrider.org/nuclear-weapons/article

277、s/could-chatbot-teach-you-how-build-dirty-bomb.129 Interview with academic,14 July 2023.130 Ibid.131 Interview with government representative,28 July 2023.132 Ibid.133 CETaS workshop participant,4 October 2023.134 CETaS workshop participant,4 October 2023.Ardi Janjeva,Alexander Harris,Sarah Mercer,A

278、lexander Kasprzyk and Anna Gausen 35 to sharing racist and antisemitic images;a“guide to memetic warfare”which advises far-right propagandists on how to use AI image tools;and the generative AI-enabled production of posters by pro-al-Qaeda outlets.135 Looking further ahead,it will be important to mo

279、nitor whether terrorist groups apply generative AI more directly to the task of persuasion for example,through conversational agents which have constant availability and limitless patience.Some researchers draw parallels with a gaming context:generative AI might enable users to develop more persuasi

280、ve narratives,characters and environments for the purpose of bolstering recruitment opportunities.136 On Thursday 5th October 2023,Jaswant Singh Chail was convicted of treason and given a 9-year sentence;Chail had broken into Windsor Castle in possession of a crossbow and declared his wish to kill t

281、he Queen.The trial heard that in the lead up to this event,Chail had exchanged over 5000 messages with an online companion named Sarai that he had created through an app called Replika.137 Many of these messages were representative of an emotional and sexual relationship;Sarai was also shown to have

282、 encouraged Chail to act out on his expressed purpose to assassinate the queen of the royal family.A University of Surrey study found that Replika tends to accentuate negative feelings that people interacting with it already have,offering an insight as to why the Sarai persona created by Chail offer

283、ed continued support and affirmation for his ability to carry out such an act.138 However,there remains a distinctly human element to the process of radicalisation which the current generation of generative AI will be unlikely to replicate.Both the literature on radicalisation and interviews for thi

284、s project emphasise that the starting point for radicalisation is predominantly through a trusted contact;it also requires traits such as 135 Tech Against Terrorism,“Early terrorist experimentation with generative artificial intelligence services,”Tech Against Terrorism Briefing,8 November 2023,http

285、s:/techagainstterrorism.org/news/early-terrorist-adoption-of-generative-ai.136 Daniel Siegel and Mary Bennett Doty,“Weapon of Mass Disruption:Artificial Intelligence and the Production of Extremist Propaganda,”Global Network on Extremism and Technology,Insights,17 February 2023,https:/gnet-research.

286、org/2023/02/17/weapons-of-mass-disruption-artificial-intelligence-and-the-production-of-extremist-propaganda.137 Tom Singleton et al.,“How a chatbot encouraged a man who wanted to kill the Queen,”BBC News,6 October 2023,https:/www.bbc.co.uk/news/technology-67012224.138 Hannah R.Marriot and Valentina

287、 Pitardi,“One is the loneliest number Two can be as bad as one.The influence of AI Friendship Apps on users wellbeing and addiction,”Psychology&Marketing(September 2023).The Rapid Rise of Generative AI:Assessing risks to safety and security 36 empathy and humour which machines currently find more ch

288、allenging to capture.139 This indicates a potential distinction between extremists using generative AI tools for the purpose of glorification rather than radicalisation.Although the two cannot be wholly separated(successful glorification tactics can have an influence on likelihood of radicalisation)

289、there is a more immediate gain for those tasked with producing and disseminating extremist content that captivates a willing audience,rather than those tasked with the next stage of convincing potential recruits to commit terrorist acts.140 Regarding radicalisation,scale and reach only go so far.In

290、some cases,the authenticity of the message being disseminated is especially important:The scale of LLM outputs is a double-edged resource.When information is abundant,attention is scarce,and being able to produce vast quantities of stuff does not always help.For Jihadists especially,authenticity mat

291、ters,so they would not easily delegate the ownership of their message to a“sexbot”.On the other hand,in the extreme right-wing terrorism context,the ownership of the message is perhaps not as important as the message itself.141 This reinforces the importance of a nuanced analysis of how different ty

292、pes of terrorist groups may engage with generative AI.Some groups may be more comfortable than others with corruptible chatbots spreading their message far and wide,even if there is a trade-off with accuracy,while other groups may prioritise more logistical or operational applications,such as using

293、generative AI to vet entry into closed groups.142 The terrorism research landscape suffers from a data deficit,because as a percentage of the population,only a very small number are radicalised by any ideology.This means developing rigorous typologies is essential prior to enacting potential legisla

294、tive responses.2.4 Weighing malicious and incidental sources of risk There remains an incompleteness to the above depiction of national security risks requiring additional analysis from two angles:first,whether the malicious risk is most pronounced from traditional state actors or from non-state act

295、ors;second,whether the lens of malicious generative AI risks is sufficient,considering the possible harms created through non-malicious incidents,mishaps,and unintended consequences.139 Interview with academic,7 July 2023;interview with government representative,18 August 2023;Alexander Babuta,“Onli

296、ne radicalisation:the need for an offline response,”RUSI Commentary,25 September 2017,https:/rusi.org/explore-our-research/publications/commentary/online-radicalisation-need-offline-response.140 Interview with government representative,18 August 2023.141 CETaS workshop participant,4 October 2023.142

297、 CETaS workshop participant,4 October 2023.Ardi Janjeva,Alexander Harris,Sarah Mercer,Alexander Kasprzyk and Anna Gausen 37 Figure 3.Sources of generative AI risk 2.4.1 State adversary and lone-actor risk From the malicious AI perspective,there are three broad categories of threat actor.The first is

298、 the state actor which may use generative AI as part of a wider armoury for targeting the UK.The second is the non-state hostile actor,such as an organised crime group,which mobilises considerable resources to undermine public safety and the rule of law.The third is most decentralised in nature lone

299、-actors who may not necessarily be affiliated with organised groups or state adversaries but are motivated to use generative AI to inflict harm.Most interviewees felt that there was not yet sufficient evidence to make confident assertions about whether state-level adversaries or lone-actors would po

300、se a greater national security risk using generative AI.However,as mentioned on numerous occasions throughout this report,there could be an additional marginal benefit to non-state actors who now have a much lower barrier to entry to the highest level of language modelling capabilities and are able

301、to operate with a level of flexibility(and fewer constraints)compared to more traditional hostile state actors.143 One specific example is the evolution of the bioweapons landscape.One interviewee cautioned against envisioning bioweapons programmes as vast Cold-War style facilities 143 Interview wit

302、h academic,7 July 2023.The Rapid Rise of Generative AI:Assessing risks to safety and security 38 sprawling across countries with multiple business units.144 Developments in automation and additive manufacturing will likely make these facilities much smaller,specialised and therefore harder to distin

303、guish from facilities which produce bio-products for commercial use.145 There may also be an analogy here to generative AI if future models trend towards being smaller,more localised,and tailored towards specific individual or community needs rather than the vast,centralised,multi-billion parameter

304、models dominating the market today.146 2.4.2 Improper adoption and unintended consequences There is a separate tranche of risks that may arise because of improper AI adoption in a range of different sectors.These incidental risks were deemed by many interviewees to be more of a threat than adversari

305、es and lone actors.In this spirit,policymakers need to adopt a broader conception of risk and security to account for the ways that day-to-day injustices or errors in the way that generative AI is used could cumulatively undermine public trust in AI.One interviewee with developer experience commente

306、d that although both(adversarial and unintended)risks are significant,the most catastrophic risks in the long-run would be likely to come from accidents rather than intentional activities.147 For some parts of the national security community,this may represent a shift in mindset when the most sophis

307、ticated weaponry and technology in the world was the preserve of a relatively minute percentage of the human population,this community was programmed to anticipate adversarial threats from those who wished to inflict harm.The proliferation of cutting-edge technology to an almost universal audience i

308、s changing this equation rapidly.148 144 Interview with academic,14 July 2023.145 Ibid.146 Interview with industry representative,19 July 2023.147 Interview with industry representative,17 July 2023.148 Interview with government representative,3 July 2023.Ardi Janjeva,Alexander Harris,Sarah Mercer,A

309、lexander Kasprzyk and Anna Gausen 39 Figure 4.Improper AI adoption in different contexts 2.4.3 Critical National Infrastructure The integration of generative AI tools into CNI was greeted by many interviewees with scepticism and apprehension.Many felt that the lack of extremely high levels of reliab

310、ility makes generative AI incompatible with safety-critical systems that necessarily require the opposite.149 In this sense,many of the recent analogies that have been drawn between AI and nuclear technology are poorly constructed:the threat is not the same()if you throw ChatGPT into a nuclear comma

311、nd and control system:the first thing is that is dumb and second,it is a nuclear threat first and foremost.A better analogy is saying that AI is to the information ecosystem or cyberspace what nuclear weapons are to the physical environment.150 There is a consensus that those at the operational leve

312、l in safety-critical industries are risk-averse by nature and accustomed to an environment with numerous layers of safeguards.Despite this,there could be cause for concern if the AI hype dominating mainstream media seeps into these contexts:sometimes you worry that the fear of missing out or the fea

313、r that we are in an AI race may lead to these models being incorporated into systems before they 149 Interview with industry representative,2 August 2023;Lei Song et al.,“Pre-Trained Large Language Models for Industrial Control,”arXiv(August 2023),https:/arxiv.org/abs/2308.03028.150 Interview with a

314、cademic,7 July 2023.The Rapid Rise of Generative AI:Assessing risks to safety and security 40 are ready.151 On the other hand,there is a risk that even if generative AI stays out of the core functioning of CNI,there are blurry distinctions with other parts of the supply chain where people are making

315、 decisions,designing documents and sending emails with the help of generative AI,which later have repercussions for CNI that are difficult to retrace.2.4.4 Public Services Outside of CNI,there are a wide range of public services seeking ways to make use of advanced technology.These include instituti

316、ons responsible for health,policing,education,pensions,and welfare.Leadership and clarity regarding areas of generative AI deployment in the public sector is essential to avoid the proliferation of a behind-closed-doors culture which ushers in a variety of subtle,structural risks.152 This was deftly

317、 summarised by one interviewee:We know these models encode biased social values and certain political leanings.If we integrate them into more and more parts of our everyday life,how we write reports or make PowerPoints,then their preferences will start to shape the way that we communicate and intera

318、ct.If the desire to avoid being seen as behind the curve comes at the cost of due diligence and effective coordination across departments,there could be additional risk in the fragmentation of procurement and deployment of generative AI systems.Some research on the negative effects of overreliance o

319、n AI systems has concluded that“users alter,change,and switch their actions to align with AI recommendations”if it is difficult to trust how an LLM has been trained,the tendency for people to adjust their behaviour based on that technology could come with serious security risks.153 Training,guidance

320、 and safeguards are explored further in Chapter 4.It Is in these scenarios where good intentions to make users in government more efficient can have adverse effects which in turn have ramifications for public trust if as a member of the public I hear just a couple of examples of things going wrong,t

321、hat will shape my attitudes in relation to institutions like the police and the courts.154 151 Interview with industry representative,2 August 2023.152 Interview with academic,7 July 2023.153 Samir Passi and Mihaela Vorvoreanu,“Overreliance on AI:literature review,”Microsoft,21 June 2022,10,https:/

322、Ibid.Ardi Janjeva,Alexander Harris,Sarah Mercer,Alexander Kasprzyk and Anna Gausen 41 Despite not being AI-specific,one high-profile public sector example which has demonstrated the dangers of unquestioning faith in technology is the British Post Office scandal.Over a 14-year period,more than 700 po

323、stmasters were prosecuted for theft and false accounting,with evidence coming principally from data produced by the flawed Horizon computerised point of sale system.155 This system determined that these individuals owed up to tens of thousands of pounds,leading to bankruptcies,prison sentences and a

324、 connection to at least one suicide.156 An independent review concluded that many of the errors might have been avoided if more robust systems and better training were in place with less reliance on old infrastructure.Cases like this serve a stark warning of what can happen when the very human fear

325、of reputational damage is combined with the embrace of new technology without being able to identify and address possible defects.2.4.5 Private sector/DIY experimentation The most decentralised form of improper adoption could come through experimentation with generative AI in private sector or DIY c

326、ontexts.The ease of accessibility will attract those who previously would not have had the means nor motive to explore use cases:if amateurs get involved in complex things because they start thinking they are being“assisted by AI”()this creates a very different landscape.157 The fear of missing out

327、on the crest of the generative AI wave will possibly cloud judgments about higher risk use cases and the rigour of checks and balances.One example of this was AI-generated books about mushroom foraging that incorrectly identified species that are safe or deadly.158 Foraging safely can require“deep f

328、act checking,curating multiple sources of information,and personal experience with the organism,none of which ChatGPT has the ability to do.”159 Many of the books on this topic found on platforms like Amazon are likely to have been written by ChatGPT yet are sold and marketed as having been written

329、by a human.160 It is easy to imagine how this type of activity could be replicated across thousands of different contexts.155 Freeths,“Post Office Scandal Averting Group Actions,”https:/www.freeths.co.uk/legal-services/business-services/post-office-scandal-group-actions/.156 Michael Pooler and Jane

330、Croft,“Bankruptcy,jail,ruined lives:inside the Post Office scandal,”Financial Times,10 September 2020,https:/ Interview with academic,14 July 2023.158 Interview with government representative,18 August 2023.159 Samantha Cole,“Generated mushroom foraging books are all over Amazon,”404 Media,29 August

331、 2023,https:/www.404media.co/ai-generated-mushroom-foraging-books-amazon.160 Ibid.The Rapid Rise of Generative AI:Assessing risks to safety and security 42 In more established industry settings,there is cause for concern regarding the potential overreliance on AI-generated code,which companies may s

332、ee as an opportunity to remove human staff.Over time,this could degrade the integrity of the whole code base,cascading vulnerabilities throughout product supply chains.161 Growing separation between company management and the code underlying their products reduces the chances of being able to accura

333、tely trace how new cyber-attacks are acclimating to AI-generated code,giving threat actors a potentially major advantage.Finally,there are a series of political and social issues which the use of generative AI can enflame,despite being neither malicious in intent nor disinformation per se.For example,earlier this year Amnesty International received criticism for using AI-generated images to demons