1、1 Artificial Intelligence in Wargaming An evidence-based assessment of AI applications June 2023 Anna Knack and Rosamund Powell Anna Knack and Rosamund Powell 2 About CETaS.4 Acknowledgements.4 Executive Summary.5 Recommendations.8 1.Introduction.9 1.1 Research aims and methodology.11 1.2 Caveats an

2、d limitations.12 1.3 Structure of this report.13 2.Progress Towards Implementing AI-enabled Wargaming.14 2.1 A renewed interest in automated wargaming.14 2.2 Case studies in AI-enabled wargaming.17 2.3 Opportunities in non-defence applications of AI.21 3.Promising Applications of AI in Wargaming.26

3、3.1 AI in narrow and specialised applications.26 3.2 High-risk,high-reward AI applications in game execution and data analysis.28 4.Technical Challenges and Requirements.31 4.1 Development of a valid methodology.31 4.2.Over-trust.32 4.3.Ensuring trustworthiness.33 4.4.Establishing causality.34 4.5.A

4、I with contextual awareness.34 4.6.Data and AI system sovereignty.35 5.Policy and Ethical Challenges.36 5.1.Accountability and liability.36 5.2.Reduction of decision-making pressure on players.36 5.3.Psychologically preparing players for human-machine teaming in wargaming.37 5.4.Efficient defence pr

5、ocurement and approvals processes.37 Artificial Intelligence in Wargaming 3 6.Conclusions and Recommendations.39 Recommendations.39 Appendix A.Long-list of AI-enabled Techniques.41 A.1 Game design.42 A.2 Game execution.45 A.3 Game analysis.49 A.4 Game logistics.52 Appendix B.Glossary.54 B.1 Wargamin

6、g lexicon.54 B.2 AI lexicon.56 About the Authors.58 Anna Knack and Rosamund Powell 4 About CETaS The Centre for Emerging Technology and Security(CETaS)is a research centre based at The Alan Turing Institute,the UKs national institute for data science and artificial intelligence.The Centres mission i

7、s to inform UK security policy through evidence-based,interdisciplinary research on emerging technology issues.Connect with CETaS at cetas.turing.ac.uk.This research was funded by the UK Defence Science and Technology Laboratory(Dstl)through The Alan Turing Institutes AI Research Centre for Defence(

8、ARC-D).All views expressed in this report are those of the authors,and do not necessarily represent the views of The Alan Turing Institute or any other organisation.Acknowledgements The authors would like to thank all those who contributed their time to participate in an interview or workshop for th

9、is project,without whom the research would not have been possible.The authors are also very grateful to Lt Col.Scotty Black and Aaron Frank,for their valuable feedback on an earlier draft of this report.Artificial Intelligence in Wargaming 5 Executive Summary This report presents findings from CETaS

10、 research undertaken on behalf of the Dstl-sponsored AI Research Centre for Defence(ARC-D),examining the potential for the application of artificial intelligence(AI)and automation in wargaming.The research focused specifically on segments of manual analytic wargames with partially simulated elements

11、.The study aimed to identify ways that AI could 1)increase the efficiency of preparing and implementing a game,2)support player decision-making and 3)improve the insights that can be gained through wargames.The findings are based on a literature review,expert interviews,case study analysis,and a wor

12、kshop convening experts across the defence and game AI community.The study explored case studies trialling the integration of AI in wargaming such as Group Ws BrainSTORM,DARPAs Gamebreaker AI,and DARPAs Stabilizing Hostilities through Arbitration and Diplomatic Engagement(SHADE)project.The study als

13、o examined non-defence applications of AI in commercial games,healthcare,and diplomacy,which may yield inspiration for segments of wargaming.Wargaming and simulation continue to be crucial tools for decision-makers in Defence.They can be used to train personnel for future conflict,and offer insights

14、 on critical decisions in warfighting,peace negotiations,arms control,and emergency response.Nevertheless,wargames also come with several challenges including the length of time required to prepare games,high set up costs,limited repeatability,and incomplete data capture within wargames.Given advanc

15、es in non-defence AI,and game AI in particular,there is growing interest in leveraging AI for wargaming and simulation.The envisaged benefits are specific to the context of use,but examples include reducing the number of personnel required,increasing the speed of development of game mechanics,improv

16、ing player immersion,speeding up game execution,and identifying innovative strategies and actions.The research identified two key features of the current landscape of AI-enabled wargaming that make it harder for decision-makers to determine whether AI can in fact achieve these benefits:1.This is a n

17、ascent debate,which has been heavily influenced by AI hype.While many ideas are circulating on how AI could improve wargaming workflows,few real-world case studies offer concrete evidence of effectiveness.Anna Knack and Rosamund Powell 6 2.AI-enabled wargaming is a subject that prompts highly varied

18、 opinions between subject matter experts.Much disagreement can be attributed to differences in expertise and experience,for example between wargamers and experts in modelling and simulation,or between technical experts and strategic decision-makers.Within this context,this report aims to advance the

19、 debate by taking an evidence-based approach to assessing the feasibility of specific AI use cases for wargaming,outlining both their risks and potential benefits.A long list of promising use cases identified is presented in the below table,with each use case described and analysed in more detail in

20、 Appendix A.Game design Game execution Game analysis Game logistics AI red team agents Procedural Content Generation Scenario and inject generation Gameplay testing Simulating opponents responses Adjudication AI to run through tactical decisions AI decision assistants and Course of Action analysis t

21、ools Virtual assistants(to provide reminders)AI to highlight interesting player interactions Data capture on player behaviour Player decision-making analysis Real-time language translation Collating past wargame data with open-source intelligence(OSINT)and secret information Speech transcription Bey

22、ond specific AI applications,this report explores two possible investment pathways for AI in wargaming:1.Narrow,specialised AI applications for the near-term:These are the simple AI use cases for near-term deployment.They include certain applications of domain-agnostic AI,for example real-time trans

23、lation or automated transcription,as well as more specialised AI applications which can be deployed with minimal risk,for example procedural content generation for background content or natural language processing for data analysis.2.High-risk,high-reward AI investments:While there were more polaris

24、ed views on advanced AI applications such as decision aids and AI for adjudication,these warrant further exploration since they have high transformative potential.However,Defence should be cautious of investing in overambitious AI-based tools before fundamental technical and policy challenges have b

25、een addressed.Several technical,policy,and ethical challenges were identified.For example,there is a need to ensure that the manner in which insights are gained from wargames can be Artificial Intelligence in Wargaming 7 understood by all relevant stakeholders.AI could hinder this,making it harder f

26、or the inner workings of the wargame to be understood.Explainable AI that can demonstrate causality between decisions and impacts must therefore be prioritised.Additionally,AI may make it possible to quickly generate a game that appears compelling,but does not represent key concepts of value to the

27、players and sponsors in a meaningful way.Efforts to identify how to build a trustworthy solution are necessary to enable adoption.Furthermore,AI that can demonstrate causality between decisions and impacts is required,as well as sovereignty discussions around the AI system and the access to the data

28、 underpinning the system.Issues around accountability and liability,as well as the potential de-skilling of senior decision-makers will also require consideration.There will be a need to psychologically prepare players for the integration of AI in wargaming.Finally,defence procurement and approvals

29、delays would need to be overcome to be able to deploy an AI-enabled system in wargaming.We conclude that the benefits AI can bring to wargaming could be significant,but there would be benefit in first introducing automation in specifically tactical or abductive wargames in the near term to manage ri

30、sks.While some narrower applications of AI(e.g.speech transcription to support data analysts)can be deployed in the near-term,the most ambitious and transformative applications(e.g.Course of Action analysis and AI adjudication)require further research and investment.Similarly,further investment in c

31、ross-cutting enablers is required before AI can be introduced effectively into strategic-level wargames.Anna Knack and Rosamund Powell 8 Recommendations The recommendations summarised below focus on preparing the UK Defence community to develop and deploy effective AI-enabled wargaming systems in a

32、responsible and evidence-based way:1.Invest in cross-cutting AI enablers The technical challenges identified in this study apply to various AI use cases,so cross-cutting enablers may be a more efficient use of resources to enable desirable applications.AI with contextual awareness,AI explainability

33、techniques or AI that can establish causality between decisions and their effects would all help to unlock several AI use cases that could bring revolutionary benefits to decision-making.2.Develop verification and validation(V&V)procedures for AI wargaming tools Experts highlighted the prevalence of

34、 non-AI automation being marketed as AI,and risks around inexpert customers procuring low-quality AI-enabled wargaming tools.Standardised V&V procedures will be important to ensure the reliability of AI tools for wargaming and their compatibility with rules of engagement,ethical standards,and explai

35、nability requirements.3.Trial narrow AI use cases in parallel to investing in complex AI-enabled wargaming There is a high cost to benefit ratio in the more sensationalised AI use cases such as CoA analysis and adjudication explored in this report.There is merit in testing implementation of narrow a

36、nd relatively low risk automation and AI use cases(e.g.Procedural Content Generation,first pass scenario generation,and speech transcription)in wargames,while simultaneously investing in research to advance complex AI-enabled wargaming(e.g.AI for player decision-making analysis and AI in adjudicatio

37、n)in parallel.4.Commission research on wargaming epistemology and decision-making in wargaming A better understanding of what constitutes an effective wargame and successful human decision-making during a wargame would enable the design of AI-enabled tools that augment rather than add uncertainty to

38、 the validity of wargame outputs.In strategic wargames,a better understanding of quantifiable parameters that affect strategic outcomes would be required.The values that decision-makers apply in their decision-making would also need to be identified.Artificial Intelligence in Wargaming 9 1.Introduct

39、ion Rapid technical advances have seen game AI make headlines globally,as AI agents have beaten human masters in ancient board games such as Go as well as strategic video games including Dota and Starcraft.1 The Chinese company Tencent has also reportedly created AI agents that have beaten human pla

40、yers at Starcraft II.2 Even complex features of human social interaction,including behaviours such as reciprocity and alliance formation,have been replicated by AI agents designed to play games like Diplomacy.3 In the UK Defence community,there is growing interest in applications of AI to wargaming

41、to help address some of the novel challenges now facing strategic decision-makers,as well as some of the enduring challenges in implementing wargames.Recent years have seen the complexity of the threat environment and the operational tempo increase rapidly,particularly in the context of sub-threshol

42、d and hybrid threats,as well as advances in emerging technologies.Consequently,UK Defence needs to grapple with overlapping,concurrent challenges,as well as novel escalation pathways within a context of uncertainty.Policymakers are now considering integrating AI in wargaming,both to prepare for the

43、potential increased speed of decision-making required in AI-supported warfighting and in order to update the wargaming methodology itself.This report considers ways in which senior decision-makers can update their analytic toolbox using AI and automation to achieve decision advantage.4 There are som

44、e early signs of government,academic,and private sector exploration of AI to assist in the design,execution,and analysis of wargames as well as to support with wargaming logistics.Moreover,applications of AI in other sectors such as healthcare,diplomacy,education,and others reveal opportunities that

45、 may be leveraged in the wargaming context.However,there is a lack of sufficient research evidence around fundamental questions including:1 James Goodman,Sebastian Risi,and Simon Lucas,“AI and Wargaming,”arXiv(September 2020):1-46.2 Peng Sun et al.,“TStarBots:Defeating the Cheating Level Builtin AI

46、in StarCraft II in the Full Game,”arXiv(September 2018):1-24.3 Tom Eccles et al.,“The imitation game:learned reciprocity in Markov games,”Proceedings of the 18th International Conference on Autonomous Agents and MultiAgent Systems(May 2019):1934-1936.4 HM Government,Defence Artificial Intelligence S

47、trategy(Ministry of Defence:2022),https:/www.gov.uk/government/publications/defence-artificial-intelligence-strategy/defence-artificial-intelligence-strategy.Anna Knack and Rosamund Powell 10 Whether AI really does support the outcomes of analytic games;Which wargaming contexts are most suited to AI

48、 deployment;What possible applications from non-defence sectors may be useful in segments of the analytic wargame workflow;Technical and policy challenges that need to be overcome to achieve the envisaged benefits that AI can bring to wargaming.This study is intended to enhance understanding around

49、the issues listed above.The study considers the whole lifecycle of an analytic wargaming,from game logistics and game design,through to execution,data capture and analysis.Figure 1 below presents a simplified illustration of this analytic wargaming workflow.Figure 1.Overview of wargaming workflow.Ar

50、tificial Intelligence in Wargaming 11 1.1.Research aims and methodology This report aims to address the following research questions:RQ1:What is the current state-of-the art in AI-enabled systems that could support different stages in analytic wargaming?RQ2:Which non-defence decision-making contexts

51、 are leveraging AI-driven technologies and may be relevant comparators for analytic wargaming in the defence context?RQ3:What potential lessons and promising practice for the future of human-machine teaming in wargaming may be derived from:o International case studies of human-machine teaming in war

52、gaming?o Relevant comparators to wargaming sub-components from other sectors and contexts?RQ4:Which AI applications for wargaming sub-components are showing the most promise?RQ5:What are the key challenges and obstacles to the introduction of AI to analytic wargaming workflows?Data collection for th

53、is study was conducted over a four-month period from December 2022 March 2023,including four core research activities:1.Literature review covering technical,academic,and policy literature on topics such as AI in wargaming,game AI,policy and ethical challenges for wargaming and AI,and best practice i

54、n non-defence sectors.2.Semi-structured interviews with 25 participants,including wargame designers,commercial game AI developers,cognitive psychologists,and modelling and simulation experts.Participants were identified using purposive,non-probabilistic sampling and interview data was analysed thema

55、tically using an inductive approach.Interviews were conducted on an anonymised,non-attributable basis.3.Case study analysis exploring case studies that trialled the integration of AI in wargaming such as Group Ws BrainSTORM,DARPAs Gamebreaker AI,and DARPAs Stabilizing Hostilities through Arbitration

56、 and Diplomatic Engagement Anna Knack and Rosamund Powell 12(SHADE)project.The study also examined non-defence applications of AI in commercial games,healthcare,and diplomacy,which may yield inspiration for segments of wargaming.4.Research workshop attended by more than 40 representatives with exper

57、tise in professional wargaming,commercial game AI,modelling and simulation,non-defence AI,AI ethics and international law.The workshop focused on prioritising AI applications for wargaming into four categories:Invest,Deploy,Track,and Disregard.Workshop data was analysed using a structured,inductive

58、approach.1.2.Caveats and limitations In light of other ongoing work on this topic,5 the focus of this study was restricted to analytic manual wargames with some simulated elements(rather than fully automated wargames).The study conclusions therefore may not be generalisable to the methodology of war

59、gaming as a whole.As emphasised throughout,the application of AI to wargaming is at an early,exploratory stage.This means that despite this report drawing on significant technical expertise,further work will be needed to scrutinise the applicability of the specific AI use cases discussed below to pa

60、rticular wargame contexts.Critically,in writing this report,the research team has navigated significant variation in opinion among research participants.In particular,between professional wargamers on the one hand whose expertise in AI was frequently limited,modelling and simulation experts,and tech

61、nical AI experts with no experience of wargaming.Participants had varied views on many fundamental topics from the definition of AI and how to categorise AI applications,to the potential utility of AI and other emerging technology across the wargaming workflow,yet there were several use cases that s

62、takeholders from different backgrounds deemed to be worth tracking if not immediately deploying.A high-level analysis of the complete list of use cases in Appendix A informs the overall findings and recommendations on how to harness a diverse range of expertise to determine best practices for AI in

63、wargaming.5 Giles Ebbutt,“British Army aims for entirely new collective training capability,”Janes,July 23,2021,https:/ Baker,“Adoption of single synthetic environment predicted to save MoD 1.3bn,with total quantitative benefits reaching over 3bn,”Business in the news,August 11,2021,https:/businessi

64、nthenews.co.uk/2021/08/11/adoption-of-single-synthetic-environment-predicted-to-save-mod-1-3bn-with-total-quantitative-benefits-reaching-over-3bn/;“Improbables Defence business awarded contract for a second year,”Improbable,December 11,2020,https:/www.improbable.io/blog/improbable-uk-strategic-comma

65、nd-sse.Artificial Intelligence in Wargaming 13 1.3.Structure of this report The remainder of this report is structured as follows.Section 2 will provide an overview of progress towards implementing AI-enabled wargaming,covering existing research and case studies,with a more comprehensive list of AI

66、applications in wargaming provided in Appendix A.Section 3 will share findings from the research workshop and give detail on which applications of AI to wargaming are recommended for near-term and long-term prioritisation.Section 4 will cover the numerous technical challenges identified in the resea

67、rch,and Section 5 will synthesise the main policy challenges,before conclusions and recommendations are offered in Section 6.Appendix A contains the long-list of AI applications with a discussion of the envisaged benefits,challenges,and the rationale for their prioritisation or de-prioritisation in

68、the defence wargaming context.Appendix B contains a glossary of the wargaming and AI terms used throughout this report.Anna Knack and Rosamund Powell 14 2.Progress Towards Implementing AI-enabled Wargaming Over recent years,interest in AI-enabled wargaming has ebbed and flowed.Now,there is an observ

69、able renewed interested in automating aspects of wargaming.This interest is largely driven by the emergence of AI applications in non-defence sectors particularly the game AI sector,which could be applied to some segments of wargaming.This section contains findings and analysis on the state-of-the-a

70、rt in introducing AI into wargaming based on a literature review,interviews,and challenge and validation through an interdisciplinary research workshop.2.1.A renewed interest in automated wargaming Interest in computer-assisted wargaming is by no means new.Interviewees described that during the 1950

71、s-60s,there was a sense that computers were growing up alongside gaming,6 and as early as the 1980s,research was already investigating how to integrate wargaming with expert systems,a key branch of AI research at that time.7 This interest in automating wargaming was described by one interviewee as c

72、yclical,rising and falling as decision-makers regularly become disillusioned with quantified approaches,but then seduced by the allure of novel technologies.8 In recent years,interest in computer-assisted wargaming has been revived.As one interviewee put it,the current moment is analogous to the 80s

73、,where there is a sense there have been technological advancements which we can return to.9 Attention has now turned to utilising the latest available AI techniques from reinforcement learning to generative AI for defence decision-making.Consensus on the current state of play was consistent:AI for w

74、argaming is in its infancy,10 there is not much going on in this space,11 and AI is not really used at this stage.12 6 Author interview with think tank wargame design expert,11 January 2023.7 Paul K.Davis and Paul Bracken,Artificial Intelligence for Wargaming and Modelling(RAND Report,February 2022)

75、,https:/www.rand.org/pubs/external_publications/EP68860.html.8 Author interview(2)with think tank wargame design expert,10 January 2023.9 Author interview with think tank wargame design expert,11 January 2023.10 Author interview with private sector modelling and simulation expert,1 February 2023.11

76、Author interview(2)with private sector modelling and simulation expert,1 February 2023.12 Author interview with think tank modelling and simulation experts,2 February 2023.Artificial Intelligence in Wargaming 15 However,there are now a few exploratory projects demonstrating nascent game AI or propos

77、ing potential AI applications for wargaming and simulation.Researchers have identified techniques such as deep learning as promising for future improvements to wargaming workflows,13 while others have pointed to using behaviour modelling for computer-generated forces,14 or to using AI techniques to

78、identify which data from real-world exercises are most relevant to wargaming outcomes.15 This work is promising but nascent.Despite the abundance of ambitious ideas,there remain significant doubts about whether any of these are 1)feasible or 2)helpful in answering decision-makers questions.Many are

79、dismissive of the potential for AI to resolve any of the pervasive challenges wargamers face.16 This scepticism came through strongly in interviews,17 which indicated that many wargamers did not support widespread AI deployment in wargaming(see Figure 2 below).Some participants suggested that basic

80、automation capabilities are frequently oversold as AI.18 Simultaneously,government customers perceived lack of understanding of the inherent limitations of AI has contributed to confusion about which AI wargaming tools hold the most potential.For example,AI performs better at pattern detection than

81、identifying causal relationships.19 AI performs better at quantitative analysis than qualitative,highly contextualised analysis.20 Together,these two points of confusion on the distinction between basic automation versus AI and on the sorts of tasks AI is suited to often result in a debate which is

82、insufficiently grounded in technical expertise and overly influenced by AI hype.At the same time,there were also optimists,and some of the AI applications discussed in Appendix A were already reportedly used by some wargame designers.For example,one interviewee discussed using ChatGPT to generate a

83、first-pass version of background 13 James Goodman,Sebastian Risi,and Simon Lucas,“AI and Wargaming,”arXiv(September 2020):1-46.14 Per-Idar Evensen et al.,“Wargaming Evolved:Methodology and Best Practices for Simulation-Supported Wargaming,”Interservice/Industry Training,Simulation,and Education Conf

84、erence(December 2019):1-13.15 Cx_Wargaming,“Wargaming the Future(attribution AI and machine learning),”Video source,June 22,2021,https:/ Jennifer McArdle and Eric Hilmer,“Effectively Integrating Technology into Wargames,”Interservice/Industry Training,Simulation,and Education Conference(2022):1-12.1

85、7 Author interview with academic wargame design expert,10 January 2023;Author interview(2)with think tank wargame design expert,10 January 2023;Author interview with think tank wargame design expert,11 January 2023;Author interview with former government wargame design expert,18 January 2023;Author

86、interview with think tank wargame design expert,23 January 2023;Author interview with think tank wargame design expert,24 January 2023.18 Author interview with private sector modelling and simulation expert,15 February 2023;Author interview(2)with think tank wargame design expert,10 January 2023.19

87、Author interview with think tank modelling and simulation experts,2 February 2023.20 Author interview with government AI technical expert,17 January 2023.Anna Knack and Rosamund Powell 16 content for their games.21 Another interviewee discussed US-based efforts which claimed slow but good progress,w

88、ith one programme reportedly seeking to build a digital wargaming environment that combines a number of AI applications that several other interviewees said are not yet possible.22 This included AI for developing scenarios,for Course of Action(CoA)analysis,for red cell behaviour development(simulate

89、d adversaries)and AI to process data from wargames.23 However,the same interviewee mentioned that these applications remain at the research stage and are not yet operational capabilities that could scale to the complexity of warfare.24 21 Author interview with academic wargame design expert,17 Janua

90、ry 2023.22 Author interview with government AI in wargaming expert,26 January 2023.23 Author interview with government AI in wargaming expert,26 January 2023.24 Author interview with government AI in wargaming expert,26 January 2023.Artificial Intelligence in Wargaming 17 Figure 2.Overview of headli

91、ne messages on AI in wargaming from interview participants.2.2.Case studies in AI-enabled wargaming There are few case studies of AI-enabled wargaming in the public domain(with even fewer outside the confines of US government-funded research),and limited evidence on the Anna Knack and Rosamund Powel

92、l 18 successes and failures of these projects.Details of the most prominent projects exploring AI-enabled wargaming are summarised below.25 This gives an initial indication of how AI may be applied to wargaming,while also illustrating the exploratory nature of current projects.Notably,details of the

93、 methodology and challenges faced by these project teams are rarely published,making it difficult for others in the wargaming and modelling and simulation community to draw lessons from their work.26 DARPAs Gamebreaker AI Exploration programme(May 2020?)27 Project overview:Project using video games

94、to study ways in which AI can be used within wargames.Aim:To understand how AI might be used to tackle multiple problems in simulated games including AI to 1)assess game balance,2)identify new tactics,and 3)test out what actions are most destabilising within video games.The ultimate aim is to use AI

95、 to create imbalanced games where one side is disadvantaged and so placed under significant stress.This is envisaged as desirable for training personnel.28 Segment of wargaming:Game design and execution Game mechanics:Nine teams were tasked with breaking two commercial video games,manipulating the g

96、ames in order to identify unexpected states within games where game dynamics did not function as expected.The aim was to design a method that could extend beyond a single video game and possibly extend to wargaming scenarios.Different teams took different approaches.The Blue Waves team attempted to

97、use autonomous game playing,interpretable AI and artificial neural networks to predict outcomes and determine why certain games were won or lost.29 Meanwhile the Northrop Grumman team planned to model and break a complex simulator environment called Command:Modern Operations.30:25 Other projects may

98、 also be of interest.These include projects developing synthetic wargaming environments(such as Improbables Single Synthetic Environment or MITREs JWARS project),projects exploring the impact AI will have on battlefield decision-making(for example,CSBAs Mosaic Warfare project,“exploiting AI and auto

99、nomous systems to implement decision-centric operations”),projects harnessing AI to improve decisionmakers understandings of complex operational environments(for example,DARPA Caudal Exploration)and other projects at the intersection of AI and wargaming which are yet to publish sufficient details in

100、 the project domain(for example,Project Voltron).26 Author interview with academic wargame design expert,10 January 2023.27“Gamebreaker AI Effort Gets Under Way,”DARPA,May 13,2020,https:/www.darpa.mil/news-events/2020-05-13.28“Gamebreaker AI Effort Gets Under Way,”DARPA,May 13,2020,https:/www.darpa.

101、mil/news-events/2020-05-13.29“DARPA Gamebreaker AI Exploration Program,”Blue Wave AI Labs,no date,https:/ Grumman Awarded DARPA Gamebreaker Contract,”Northrop Grumman News Releases,August 12,2020,https:/ Intelligence in Wargaming 19 Group Ws BrainSTORM(September 2018?)31 Project overview:A DARPA spo

102、nsored project based on STORM,the US Department of Defence model for multi-domain operations at the campaign level.32 Aim of integrating AI:To assess whether AI can enhance course of action analysis and whether AI-assisted non-expert players can beat expert teams in a wargame.33 Segment of wargaming

103、:Game execution Game mechanics:BrainSTORM consists of three distinct tools:1)game cloud is oriented towards facilitating high level strategy development,2)color compare facilitates the specification of these strategies to detailed CoA plans,3)speedSTORM is the environment in which detailed AI recomm

104、endations are accessed.Evidence of effectiveness:Researchers reported that AI supported non-expert players can beat expert players.More tests are planned for the future.DARPAs Stabilizing Hostilities through Arbitration and Diplomatic Engagement(SHADE)project(January 2022 June 2023)34 Project overvi

105、ew:This programme aims to use AI to improve strategic decision-making for diplomacy,in particular looking at negotiation processes.Specifically,a simulated environment of the classic game Diplomacy is used to train and evaluate AI agents.Aim:To investigate the extent to which AI can emulate human di

106、plomatic behaviours including deception,collusion,profiling and complex multi-party interactions.Ultimately,the hope is this will improve understanding of the complex communication involved in human negotiations.35 Segment of wargaming:Game execution Game mechanics:This project is based on an open-s

107、ource Diplomacy game engine which facilitates humans to play against Diplomacy bots.The team have created benchmark bots to assist in training new SHADE AI bots.31 Alec Barker,“BrainSTORM:exploring artificially intelligent COA development in STORM,”SPIE.Library(April 2021).32 Multi-domain operations

108、 are those which span multiple domains,to include air,cyber/electromagnetic,maritime,land and space.Read more:HM Government,Multi-Domain Integration(Ministry of Defence:2022),https:/www.gov.uk/guidance/multi-domain-integration.33 Alec Barker,“BrainSTORM:exploring artificially intelligent COA develop

109、ment in STORM,”SPIE.Library(April 2021).34“Stabilizing Hostilities through Arbitration and Diplomatic Engagement,”Project homepage,SHADE,no date,https:/www.shade-aie.org/.35 There are significant parallels between this work taking place in the context of defence wargaming and work by DeepMind within

110、 game AI more broadly where researchers have studied the performance of AI agents in the board game of Diplomacy.See:Janos Kramar et al.,“Negotiation and honesty in artificial intelligence methods for the board game of Diplomacy,”Nature Communications 13(2022).Anna Knack and Rosamund Powell 20 DARPA

111、s Constructive Machine-learning Battles with Adversary Tactics(COMBAT)(May 2020?)36 Project overview:This programme aimed to develop AI applications which could model red team behaviour and be used to challenge blue teams in subsequent war games.Aim:To develop several feasible AI-generated courses o

112、f action for the red team.And,subsequently to use these AI-based CoAs to determine optimum responses from the blue team.Ultimately,the aim is to use AI to help stimulate novel blue team behaviours.Segment of wargaming:Game execution Game mechanics:As part of this project,DARPA selected companies to

113、develop AI applications and determine their own methodologies.Companies were tasked with developing their initial tactics according to“The Russian Way of War:Force Structure,Tactics and Modernization of Russian Ground Forces.”DARPAs Strategic Chaos Engine for Planning,Tactics,Experimentation and Res

114、iliency(SCEPTER)(January 2022?)37 Project overview:A project using automation and AI to explore the development of novel course of action plans at machine speed.Aim:This project aimed to investigate whether the speed of CoA planning can be increased through advanced automation and whether these CoAs

115、 can compete with human plans.Segment of wargaming:Game execution Game mechanics:The strategies developed for machine-generated CoA planning will be tested within trusted simulated environments where they will be assessed by humans.Insights from these defence case studies are summarised in Table 1 b

116、elow.Some of the case studies were based within a fully digitised platform so the results were not indicative of possibilities for purely manual wargames or manual wargames with some digital elements.Moreover,there is limited evidence in the public domain of these projects findings beyond 36“Constru

117、ctive Machine-learning Battles with Adversary Tactics(COMBAT)(Archived),”DARPA,no date,https:/www.darpa.mil/program/constructive-machine-learning-battles-with-adversary-tactics;John Keller,“Researchers eye embedding artificial intelligence into war games simulation to beef-up challenges,”Military+Ae

118、rospace Electronics,June 3,2020,https:/ John Keller,“DARPA SCEPTER project seeks to develop battle planning for complex military engagements at machine speed,”Military+Aerospace Electronics,January 20,2022,https:/ Editor,“DARPA SCEPTER Program Will Explore Machine-Generated Strategies For Warfare,”D

119、efense Techconnect:The innovation source for national security,January 24,2022,https:/ Intelligence in Wargaming 21 self-reported claims on performance which are difficult to verify and to generalise.Nevertheless,some emerging lessons may be learnt.Table 1.Lessons learnt from existing case studies o

120、f AI-enabled wargaming.Project Key findings Gamebreaker AI Indicative of possibility of learning from commercial games industry successes,rather than starting from scratch for defence applications.Highly quantified approach may be applicable to tactical games or games with a pre-determined rulebook.

121、Unclear whether this is applicable to strategic wargames with high uncertainty.BrainSTORM Early experiments show users,even non-experts,prefer to interpret AI recommendations themselves rather than adopting them wholesale.38 SHADE Indicates efforts are being made to understand how AI agents might de

122、al with human concerns at the more strategic end of the spectrum for example,deceptive human behaviours.Demonstrates the exploratory nature of much existing work,with the aim being to develop a prototype AI agent rather than a near-term capability.DARPA COMBAT Illustrates benefits of competitive res

123、earch models to drive innovative ideas within private sector.Unclear if this project demonstrates a successful application of AI red team agents given the lack of public evidence of which AI methodologies were applied or what results were achieved.DARPA SCEPTER This project is at an early stage and

124、further developments should be tracked for further insights.2.3.Opportunities in non-defence applications of AI This study also investigated where wargaming may take inspiration from non-defence sectors.While non-defence AI use cases may appear alluring,it is necessary to bear in mind three key cave

125、ats before drawing conclusions based on these applications:38 Alec Barker,“BrainSTORM:exploring artificially intelligent COA development in STORM,”SPIE.Library(April 2021).Anna Knack and Rosamund Powell 22 1.The defence context comes with specific requirements of AI assurance given the grave consequ

126、ences of any errors in decision-making,39 and given the number of novel situations which may be encountered.40 2.Wargames are highly complex and human centric.This complexity is often not reflected in other sectors(even in complex games such as Diplomacy).41 3.Budgetary constraints differ enormously

127、 by sector.The cost to reward ratio required to make defence investment worthwhile will differ from other commercial sectors.42 Nevertheless,certain lessons can be learnt from other sectors,both in cases where AI applications from elsewhere may be adapted for Defence and in cases where challenges fa

128、ced elsewhere are indicative of similar barriers faced by Defence.When asked which sectors would be most relevant to advancing AI-enabled wargaming,interview participants mentioned a wide range of sectors(as illustrated in Figure 3 below).Figure 3.Most frequently identified comparator sectors based

129、upon interview data,illustrating that healthcare and the commercial games industry were most frequently cited by interviewees as relevant to defence applications.39 Author interview with private sector modelling and simulation expert,15 February 2023.40 Author interview with government wargaming exp

130、erts,14 December 2023.41 Author interview with think tank wargame design expert,23 January 2023;Author interview with private sector modelling and simulation expert,1 February 2023.42 Author interview(2)with AI technical expert,11 January 2023.Artificial Intelligence in Wargaming 23 In addition to t

131、hese varied sectors identified in interviews,further relevant sectors for Defence to track have been identified through literature review.These include:Applications of AI in insurance,where progress has been made towards using chatbots to provide targeted information to customers;43 Commercial risk

132、management,where AI has helped to develop dashboards which summarise up-to-date global events;44 Applications in the education sector where algorithms used to grade students may provide insights for those wishing to use AI in player decision-making analysis.45 All these AI use cases and more have in

133、fluenced and inspired the applications of AI to wargaming discussed in Appendix A.However,three sectors were particularly prominent and the relevance of each to AI-enabled wargaming is summarised below.For each sector we indicate significant AI use cases drawn from relevant academic literature,while

134、 also pointing to their specific relevance to AI-enabled wargaming.46 Table 2.Relevant AI applications from the healthcare sector.AI use cases Relevance to AI-enabled wargaming AI to assist clinicians with triage decisions in emergency care47 AI for decision support in healthcare may be indicative o

135、f technical possibilities for similar applications to wargaming,in particular promising AI use cases to support triage decisions by predicting the likelihood of severe deteriorations in patient condition.48 Similar techniques may be applied to course-of-action analysis,as this is also an environment

136、 where AI assists humans in choosing which actions are favourable in a high-stakes context based on predictions of possible associated risks.43 Benno Keller,“Promoting responsible artificial intelligence in insurance,”The Geneva Association(April 2020).44 Fahim K Sufi,“AI-GlobalEvents:A Software for

137、 analyzing,identifying and explaining global events with Artificial Intelligence,”Software Impacts 11(February 2022):1-5.45 Victor Gonzalez-Calatayud,Paz Prendes-Espinosa and Rosabel Roig-Vila,“Artificial Intelligence for Student Assessment:A Systematic Review,”Applied Sciences 11,no.12(June 2021):5

138、476.46 Given the breadth of sectors addressed here,AI use cases will necessarily be discussed in broad terms and further work will be needed to explore how lessons can be learnt from specific projects from outside of Defence.47 Marta Fernandes et al.,“Clinical Decision Support Systems for Triage in

139、the Emergency Department using Intelligent Systems:a Review,”Artificial intelligence in medicine(November 2019);Aidan McParland and Kiran Grant,“Applications of artificial intelligence in emergency medicine,”University of Toronto medical journal 96,no.1(January 2019).48 Aidan McParland and Kiran Gra

140、nt,“Applications of artificial intelligence in emergency medicine,”University of Toronto medical journal 96,no.1(January 2019).Anna Knack and Rosamund Powell 24 AI chatbots in psychology and decision support tools to guide psychologists or psychiatrists during therapy sessions49 Research on AI in ps

141、ychology points to numerous ethical and policy challenges associated with decision support tools,from a persistent lack of trust to cognitive de-skilling of practitioners.The defence sector can draw on this work for virtual assistants that provide legal reminders or CoA.Table 3.Relevant AI applicati

142、ons from the commercial game sector.AI use cases Relevance to AI-enabled wargaming Deep learning for procedural content generation(PCG)50 Progress towards using deep learning for PCG is significant to wargame-design as the sophistication of AI-designed game elements,such as non-player characters,is

143、likely to increase.AI game agents,including for games such as Dota,Starcraft,and Go51 AI game agents should be tracked by those interested in using AI to automate red or blue teams in wargaming.However,the decision space of these games is typically narrower,meaning lessons are not directly transfera

144、ble.Additionally,the end goal of this research is often simply to create human-beating AI agents,whereas in wargaming,greater emphasis is placed on the insights gained by analysing human actions within the game itself.AI for the boardgame Diplomacy52 Certain complex games show a higher relevance to

145、the context of wargaming,with DeepMind research on Diplomacy showing particular promise given progress towards emulating human behaviours such as negotiation and alliance formation.53 Results show promise with regard to AI agents abilities to act both cooperatively,forming alliances,and duplicitousl

146、y,breaking alliances.49 Christopher Burr and Rosamund Powell,“Trustworthy Assurance of Digital Mental Healthcare,”The Alan Turing Institute(November 2022).50 Jialin Liu et al.,“Deep Learning for Procedural Content Generation,”arXiv(Ovtober 2020).51 David Silver et al.,“Mastering the game of Go with

147、deep neural networks and tree search,”Nature 529(January 2016):484-489;Christopher Berner et al.,“Dota 2 with Large Scale Deep Reinforcement Learning.”ArXiv(December 2019);Dan Garisto,“Google AI beats top human players at strategy game StarCraftII,”Nature News(October 2019);Janos Kramar et al.,“Nego

148、tiation and honesty in artificial intelligence methods for the board game of Diplomacy,”Nature communications 13(December 2022):7214.52 Janos Kramar et al.,“Negotiation and honesty in artificial intelligence methods for the board game of Diplomacy,”Nature communications 13(December 2022):7214.53 Jan

149、os Kramar et al.,“Negotiation and honesty in artificial intelligence methods for the board game of Diplomacy,”Nature communications 13(December 2022):7214.Artificial Intelligence in Wargaming 25 Table 4.Relevant AI applications from diplomacy and conflict prevention.AI use cases Relevance to AI-enab

150、led wargaming Research has explored AI for decision support in the context of diplomacy,for instance using AI to provide recommendations to ministries on strategic decisions.54 Work on AI decision support for diplomacy will be of particular relevance to those wishing to use AI to provide virtual ass

151、istants for wargame players,which consider complex international political environments.For example,tools to remind players of the legal context relevant to a particular game.AI to analyse transcripts during diplomatic negotiation.55 AI for analysing transcripts in the context of armed conflict prev

152、ention is indicative of the potential of AI to assist in analysis of wargaming transcripts,particularly to analyse convergence and divergence in viewpoints between teams over the course of a wargame.This overview of progress towards AI-enabled wargaming illustrated that while significant progress ha

153、s been made in adjacent sectors,much of this is as yet untapped within wargaming itself.As the remaining sections will reiterate,further research is needed to assess the extent to which existing AI techniques may be practical within wargaming workflows and the extent to which they can help to addres

154、s some of the persistent challenges of analytic wargaming.54 Corneliu Bjola,“Diplomacy in the Age of Artificial Intelligence,”EDA Working Paper(January 2020).55 Author interview with AI technical expert,10 January 2023;Miguel Arana-Catania,Feliz-Anselm van Lier and Rob Procter,“Machine Learning for

155、Mediation in Armed Conflict,”arXiv(August 2021).Anna Knack and Rosamund Powell 26 3.Promising Applications of AI in Wargaming Despite broad variance of opinion on the utility of AI in wargaming,and a lack of consensus on which AI applications hold the most utility,even the more cynical interview par

156、ticipants identified particular contexts in which AI may bring benefits to wargaming.Figure 4 below presents the longlist of AI use cases identified,which are discussed in further detail in Appendix A.This section only focuses on the list of use cases prioritised in a March 2023 workshop.Figure 4.Lo

157、ng-list of AI use cases for wargaming identified AI was assessed to be useful in narrow and more specialised applications like tactical gameplay and abductive games,as well as AI applications for supporting data analysis where the main risks to the quality of gameplay can be managed.Abductive games

158、are games with a known end state where the objective of the game is to identify the best possible explanation for a surprising fact or observed phenomenon.Tactical and abductive games lend themselves more to the integration of AI because the possible outcomes and impacts of decisions are more often

159、known and quantifiable.There were also some high-risk,high-reward applications mentioned.These would require significant further investment and research,but some study participants believed they could contribute to breakthroughs in decision advantage through wargaming.3.1.AI in narrow and specialise

160、d applications The most readily deployable AI solutions do not alter integral game mechanics,but could instead improve the logistics of the game,support data analysis,or add more realistic texture and noise to the game(e.g.,synthetic Tweets or diplomatic cables that add tension to the game).Speeding

161、 up processes like automated speech transcription and language Artificial Intelligence in Wargaming 27 translation during a wargame using automation,or using generative AI for a first draft of scenarios was considered by interviewees to present relatively low risk.56 Often,these more practical appli

162、cations were not at the cutting edge of AI research,and several interviewees cited simpler non-AI automation techniques as most useful.57 However,while simple automation could provide significant efficiency gains,a preoccupation with falsely marketing these automation techniques as AI has contribute

163、d to hype and confusion.58 Greater technical literacy is needed to distinguish between AI and simpler automation techniques,and to identify which option is best applied to a particular wargaming context.We need more mini models rather than large command rooms with complex simulations.This could brin

164、g benefits of modelling to a great variety of roles in wargaming.59 One of the reasons for the limited progress in integrating AI into wargaming is that ambitions for these systems are too high,with visions of automating all aspects of wargaming from game preparation through to execution and analysi

165、s.Instead,AI development for wargaming should be targeted for specific purposes.60 For example,in data capture and analysis for analytic wargames,interviewees highlighted opportunities to reduce analysis time by increasing the efficiency of data capture on what transpired in the game,or collating wa

166、rgame data,secret information and open-source intelligence.Introducing AI to take over tactical decisions could also potentially reduce the number of personnel required in a wargame.In a strategy game where the effect is distant from the cause,and complex battlespaces are orders of magnitude larger,

167、it would be more difficult to envision reliable AI.AI is best suited to making relatively simple decisions such as firing policy,where cause and effect are seconds apart and the simulation can be run iteratively.Moreover,the more abductive,quantifiable,and rules based the game is,the more likely AI

168、might be able to help in a simulated environment.61 The more strategic the game,the fuzzier 56 Author interview with private sector modelling and simulation expert,15 February 2023;Author interview with think tank wargame design expert,10 January 2023.57 Author interview with private sector modellin

169、g and simulation expert,15 February 2023;Author interview with think tank modelling and simulation expert,17 January 2023;Author interview with private sector modelling and simulation expert,1 February 2023.58 Author interview with private sector modelling and simulation expert,15 February 2023;Auth

170、or interview with think tank modelling and simulation expert,17 January 2023;Author interview with private sector modelling and simulation expert,1 February 2023.59 Author interview with government AI technical expert,19 January 2023.60 Author interview with think tank wargame design expert,24 Janua

171、ry 2023.61 Author interview with private sector modelling and simulation expert,1 February 2023.Anna Knack and Rosamund Powell 28 the causality becomes,leaving many aspects of the game open to interpretation and unknown outcomes.As one interviewee stated,If AI can solve it for you,you dont need a wa

172、rgame,you could just run a simulation.62 This illustrates the importance of clarifying the aim of a wargame before considering the introduction of AI.If a wargame is being used to understand other human adversaries decision-making,AI resolving the problem for the player will not bring the same sorts

173、 of insights as humans tackling the problem themselves.However,if the game is seeking to understand how to compete with human adversaries who are assisted by AI-enabled CoA analysis,then some of the more complex AI use cases in Section 3.2 may be worth exploring.3.2.High-risk,high-reward AI applicat

174、ions in game execution and data analysis There were a few bleeding-edge AI applications that interviewees believed could constitute revolutionary innovation to the type of data that can be extracted from a wargame.These include decision aids that support course-of-action(CoA)analysis potentially thr

175、ough virtual assistants and chatbots,AI in adjudication,and AI that enables analysis of player decision-making through a combination of computer vision and Natural Language Processing(NLP).These revolutionary applications could be bending the possible definitions of a wargame itself and the potentia

176、l benefits could be significant.However,these potentially high reward applications were also the most disputed,and some warned that developing AI tools for these applications could be the most expensive and least effective way to pursue decision advantage.63 Some described this challenge as intracta

177、ble,especially given the sub-optimal results emerging from US projects,despite such significant investments.64 However,though this innovation approach may necessitate a long road of failures before these use cases are successfully demonstrated,investment in these use cases could also generate revolu

178、tionary advances that constitute a competitive edge in strategic competition.62 Author interview with think tank wargame design expert,24 January 2023.63 Author interview with think tank wargame design expert,23 January 2023.64 Author interview with think tank wargame design expert,11 January 2023.A

179、rtificial Intelligence in Wargaming 29 AI for CoA analysis and other decision aids could help speed up decision-making,or help players consider novel tactical options and present their likely results.65 There are some emerging projects such as DARPAs Strategic Chaos Engine for Planning,Tactics,Exper

180、imentation and Resiliency(SCEPTER)project,which seeks to explore the use of decision aids producing machine-generated strategies to compete with humans in simulated environments,but little is known of the concrete results.66 In order to calibrate the right level of trust in AI-enabled CoA,there is a

181、 need for players,game sponsors and overseers to understand how AI has arrived at its outputs,but the state-of-the-art in AI is insufficiently transparent to enable this.Players and data scientists seeking to understand the games also need to understand why they succeeded or failed.67 Moreover,the d

182、ata truly needed to bring value to a wargame includes insights such as which CoAs were discarded and for what reason.This may be challenging for AI-enabled CoA,which may not be able to fully explain its reasoning.68 Adjudication is one arena in which expert viewpoints diverged significantly,with som

183、e suggesting machine learning might be effectively leveraged to determine outcomes in novel situations,while others warned strongly against this.Adjudication is usually done by developing a results table of known outcomes and rolling the dice,and there is interest in integrating AI to make adjudicat

184、ion less random and more predictive based on historical analysis.69 But this only works well if the potential consequences of players actions are well known and understood.70 If the game in question is an analytic game aimed at understanding the potential decision-making of an adversary in a novel s

185、ituation,then everyone is equally inexpert,and by definition,there will not be a results table that can be programmed into the AI.71 Moreover,computerised adjudication is relatively inflexible if new insight is revealed throughout the wargame,requiring redesign of the scenario or rules of the game.7

186、2 In manual wargames,humans with expert knowledge can quickly adapt the rules,scenario and the next script of events to the decisions of players.This does not mean that human adjudication is perfect,but instead that any plans to integrate AI will need to 65 Author interview with academic wargame des

187、ign expert,30 January 2023;Author interview with government AI in wargaming expert,26 January 2023.66 John Keller,“DARPA SCEPTER project seeks to develop battle planning for complex military engagements at machine speed,”Military+Aerospace Electronics,January 20,2022,https:/ Author interview with th

188、ink tank wargame design expert,24 January 2023.68 Author interview with think tank wargame design expert,24 January 2023.69 Author interview with government AI technical expert,19 January 2023.70 Author interview with academic wargame design expert,30 January 2023.71 Author interview with academic w

189、argame design expert,30 January 2023.72 Author interview with think tank wargame design expert,24 January 2023.Anna Knack and Rosamund Powell 30 ensure the sort of continued flexibility in adjudication outcomes which can currently only be achieved with expert involvement.Another challenge is that fo

190、r both CoA analysis and adjudication,there are innumerable possible environmental factors,and no existing datasets to train AI on all possible outcomes by several potentially illogically behaving agents.Artificial Intelligence in Wargaming 31 4.Technical Challenges and Requirements This section exam

191、ines the key technical challenges and requirements for effectively integrating AI into wargaming based on inputs from the literature review,interviews and workshop.These include the intricacy of developing a valid methodology,risk of users over-trusting AI system outputs,ensuring trustworthiness,est

192、ablishing causality,providing AI systems with an appropriate degree of contextual awareness and considerations around AI system and data sovereignty.These challenges and requirements are each considered in turn.4.1.Development of a valid methodology Constructing a wargaming methodology that yields l

193、ogically valid outputs is an intricate art.Introducing AI in a way that does not further add doubt to the validity of the outputs of the analytic wargame will be crucial to the adoption of AI in wargaming.Humans are not rational decision-makers,and when humans are involved,game theory rarely plays o

194、ut in practice and results are always different.This led a wargame designer to suggest it would be difficult for AI to replicate human decision-making patterns in AI red team agents or in predictive analytics,73 particularly without labelled datasets on adversary decision-making patterns.74 While th

195、e emergent behaviour of AI may appear complex,it may be derived from only basic rules,whereas human behaviour can be creative and unpredictable.75 This perceived inability to mimic the emergent behaviour of humans is seen as a disadvantage by many when the purpose of the wargame is to improve capabi

196、lities against humans,not machines.76 Some envisaged that reinforcement learning may mitigate this need for data or help enumerate every possibility in ways humans cannot,and that AI creativity could advance in ways that overcome these limitations.It should be noted,however,that research in this are

197、a is nascent.In AI techniques like agent-based modelling,it is important to model features of the environment and if all the relevant factors are not modelled,then the agent behaves in ways that do not make sense and therefore make the results of the wargame invalid.77 In manual 73 Author interview

198、with government wargaming experts,10 December 2022.74 Author interview with private sector modelling and simulation expert,15 February 2023.75 Author interview with private sector modelling and simulation expert,15 February 2023.76 Author interview with government AI technical expert,17 January 2023

199、.77 Author interview with private sector modelling and simulation expert,15 February 2023.Anna Knack and Rosamund Powell 32 wargames,it is possible to have human players replicating the decisions of hostile states,but even experts will disagree on whether these types of wargames reflect the ground t

200、ruth.One interviewee emphasised:Wargaming has poorly defined error bars.Its inaccurate because its making some predictions about scenarios and the end point is never going to happen because there are so many decisions that would need to be made in the real world.78 Some interviewees highlighted that

201、 there may be a temptation for commercial wargame suppliers who lack the background in wargame design to rush to develop the code for AI in wargaming before the validity of the methodology is established.79 In order to overcome this challenge,more research is required to understand how theoretical m

202、odels behind wargames can be integrated with AI and still be intellectually defensible.This means setting definitions that can be coded for nebulous concepts like democracy80 and quantitatively optimising for these definitions.At the same time,there is a potential trade-off between optimal,yet britt

203、le and predictable solutions with set definitions and more robust solutions that allow the AI to explore without being constrained by narrow definitions.The key will be knowing the right contexts to prioritise optimal or robust solutions.These risks could be further exacerbated by government custome

204、rs limited technical understanding of AI.One interviewee with experience trialling the integration of AI in a simulation said he felt like the wizard of Oz behind the curtain,because what players in the wargame assumed the computer did was far more complex and sophisticated than what the computer ac

205、tually did.81 We dont have the foundations,but we are trying to build the cathedral anyway.82 4.2.Over-trust The epistemological validity of the outputs of a wargame and the capacity of different types of AI may be variably understood across UK Defence,especially at the senior levels.83 When players

206、 are immersed,they may stop critically assessing the model and place too much trust 78 Author interview with think tank wargame design expert,24 January 2023.79 Author interview with think tank wargame design expert,23 January 2023.80 Author interview with think tank wargame design expert,23 January

207、 2023.81 Author interview with private sector modelling and simulation expert,15 February 2023.82 Author interview with think tank wargame design expert,23 January 2023.83 Author interview with think tank wargame design expert,10 January 2023;Author interview with think tank wargame design expert,23

208、 January 2023.Artificial Intelligence in Wargaming 33 in the outputs.84 An illustrative example is in AI-enabled red team agents,which simulate the actions of opponents,but which are difficult to validate for realism.85 Another example is in generative AI for scenario generation,which may not be bas

209、ing its outputs on a sufficiently credible or robust evidence base.86 Immersed players may forget to challenge the validity of AI outputs.Wargames are not meant to be exact predictions and traditionally adjudicated wargames are not always challenged either,but AI may add further uncertainty to the d

210、efensibility of wargame outputs.In order to develop a wargame with valid outputs,the theoretical model underpinning the game needs to be clear to policymakers and developers.87 There is therefore a need to cultivate an awareness that depending on different epistemological positions,the resulting ben

211、efits of a wargame or simulation are considered differently.4.3.Ensuring trustworthiness Even if the technical barriers to some of the AI techniques discussed in the previous section were to be overcome,there would remain several ethical,legal and political challenges which would need to be overcome

212、 in order to engender widespread trust in decisions made on the basis of AI-enabled wargames.Some interviewees believe that integrating AI in wargaming is far enough from high-risk decisions to be relatively low risk,88 but others contended that there are still ethical risks around taking conclusion

213、s from an oversimplified synthetic decision-making context and applying negative learning in the operational environment outside the wargame.89 On the other hand,it would be near impossible to determine whether it was the wargame or other factors that affected a senior decision-makers decision-makin

214、g.Furthermore,it is unclear how ethics can be encoded in some use cases suggested,such as virtual assistants that provide reminders on ethical considerations,since sufficiently ethical actions in every given context are subjective and not easily quantifiable thresholds that can be coded into a machi

215、ne.90 84 Author interview with government AI technical expert,19 January 2023.85 Author interview with academic wargame design expert,18 January 2023,Author interview with government AI technical expert,17 January 2023.86 Author interview(2)with AI technical expert,11 January 2023.87 Author intervie

216、w with think tank wargame design expert,23 January 2023.88 Author interview with government wargaming experts,10 December 2022.89 Author interview with think tank wargame design expert,11 January 2023;Author interview(2)with private sector modelling and simulation expert,1 February 2023.90 Author in

217、terview with government AI technical expert,19 January 2023.Anna Knack and Rosamund Powell 34 Moreover,some stakeholders anticipated difficulty in securing senior decision-makers buy-in for AI-enabled systems that could invasively collect and analyse data on their decision-making performance,particu

218、larly because simulated models have shown low reliability to date.91 4.4.Establishing causality The main issue that makes it difficult to transform strategic level wargaming using AI applications is the difficulty of predicting the causal chain of events between decisions and outcomes at the strateg

219、ic level.Many of the AI techniques raised by interviewees(see Appendix A for a comprehensive list),from natural language processing to reinforcement learning,are more proficient at finding correlations,92 and fuzzier at establishing the causal effects of decisions.93 Developing AI that can understan

220、d and apply the rules of the strategic context to predict these causal chains at the complexity of strategic level decisions,including escalatory and de-escalatory moves,would be a game-changing capability.94 4.5.AI with contextual awareness In order to enable human-machine teaming in AI use cases l

221、ike CoA analysis and adjudication,the human and the AI tool would need to have the same situational awareness about what is happening on the manual boardgame if the whole system is not digitised.95 Moreover,to realise the envisaged benefits of use cases like AI highlighting interesting interactions

222、between red and blue human agents to data scientists,96 AI would need to be able to recognise the most salient interactions of the players.There is ongoing,nascent research on informing AI about the context of moves in a wargame,but this is currently not yet possible in all contexts.97 There will be

223、 a need to invest in human-machine interfaces that enable players and their deliberations and actions to be integrated into the digital space where AI can process it in a fully simulated environment.91 Author interview with private sector modelling and simulation expert,15 February 2023.92 See Appen

224、dix B for further details on the AI techniques seen as most relevant to AI-enabled wargaming.93 Author interview with think tank modelling and simulation experts,2 February 2023.94 CETaS Workshop,16 March 2023.95 CETaS Workshop,16 March 2023.96 Author interview with government wargaming experts,10 D

225、ecember 2022;Author interview(2)with think tank wargame design expert,10 January 2023.97 Author interview with private sector modelling and simulation expert,1 February 2023.Artificial Intelligence in Wargaming 35 4.6.Data and AI system sovereignty If the skilled talent to develop AI-enabled wargami

226、ng systems exists in the commercial sector or in other nations,then the data and tools underpinning the systems could enable other nations and non-government actors to access sensitive information.98 This may run contrary to the pursuit of strategic autonomy on AI for strategic advantage and could g

227、enerate security risks.However,in order to build sovereign AI capabilities,there may be a need to collate large volumes of data across national boundaries,which some nations are addressing by coordinating at a European level.The UK will need to consider its own approach to data and AI sovereignty in

228、 relation to military AI and wargaming specifically.98 Author interview with think tank modelling and simulation experts,2 February 2023.Anna Knack and Rosamund Powell 36 5.Policy and Ethical Challenges This section discusses the policy and ethical challenges identified in the research which must be

229、 overcome in order to implement effective AI tools in wargaming.The analysis in this section is based on inputs from the literature review,interviews and interdisciplinary workshop.5.1.Accountability and liability The first policy challenge identified in the research relates to accountability of dec

230、ision-making.Decisions made in defence contexts may have life-or-death consequences.At the extreme end of the spectrum,if it is impossible to determine whether AIs integration in CoA analysis affects high risk decision-making,there is a risk that the decision-making informed by AI could cause loss o

231、f life,but that the AIs influence on this outcome cannot be proven or challenged.Such CoA systems should therefore not be deployed until the causality between the experience of AI in wargaming and its effects on subsequent decision-making are better understood.5.2.Reduction of decision-making pressu

232、re on players Another policy concern is whether introducing AI in wargaming or in other decision aids could have a detrimental effect on senior decisionmakers decision-making performance due to the reduction of pressure in the game environment.99 The introduction of AI use cases like virtual assista

233、nts that provide reminders on legal and regulatory frameworks,historical data or past wargames may reduce the risk of cognitive overload,100 but may also create cognitive dependency on the tools.At the same time,one interviewee contended that there is such a shortage of personnel in Defence that the

234、 risk of not trying to leverage AI is more pressing than the risk of cognitive deskilling.101 99 Author interview with academic wargame design expert,10 January 2023.100 CETaS Workshop,16 March 2023.101 Author interview(2)with think tank wargame design expert,10 January 2023;Author interview with th

235、ink tank wargame design expert,11 January 2023;Author interview with academic wargame design expert,17 January 2023.Artificial Intelligence in Wargaming 37 5.3.Psychologically preparing players for human-machine teaming in wargaming One possible way of helping humans build calibrated trust in a huma

236、n-machine teaming system in wargaming is by trialling AI applications in narrow applications to increase players experience with AI.This process could begin with automating simple manual processes,before gradually introducing more complex AI.102 Experience is the antidote to mysticism.The more the s

237、ervices get their hands on and experiment with these tools,the less they will be this mythical thing on a pedestal.103 As AI is integrated in hybrid and partially simulated manual wargames,there will be a need to determine the appropriate roles for humans and AI.The humans role will likely be more c

238、entred on situational awareness,observing how well AI is undertaking its tasks.Introducing AI in innocuous use cases like real-time language translation in multi-national wargames,then gradually introducing more advanced AI applications like AI for analysis of player decision-making patterns could h

239、elp facilitate this transition.5.4.Efficient defence procurement and approvals processes An AI-based system for wargaming will need to be approved through slow bureaucratic approvals processes and be able to navigate approvals for access to data from classified and international networks,104 which w

240、ill all generate delays.For example,interviewees discussed the utility of using AI to collate past wargame data with secret intelligence and OSINT to automate risk analysis(e.g.,via artificial neural networks),105 but approvals for all these types of data will likely create delays to model developme

241、nt.The problem you run into is for it to be genuinely labour saving it needs to be intuitive enough and work within the hellscape of any defence departments IT system.By the time you are navigating classified networks,international networks,102 Author interview with think tank wargame design expert,

242、23 January 2023.103 Author interview with private sector modelling and simulation expert,15 February 2023.104 Author interview with think tank wargame design expert,24 January 2023.105 Author interview with government AI technical expert,17 January 2023.Anna Knack and Rosamund Powell 38 different mi

243、nistries,and departments there is a possibility that a tool will be more trouble than it is worth.106 This challenge is not unique to AI,and there are ongoing efforts to address these procurement challenges across UK Defence.But if these delays are not addressed,even an intellectually defensible and

244、 technically feasible AI system may not be delivered at the point of need weakening the UKs ability to secure strategic advantage over its adversaries.Finally,appropriate upskilling of procurement professionals will be required to ensure those procuring AI systems for defence can meaningfully scruti

245、nise the systems capabilities and limitations.Professional wargamers suggested that often,one of the largest causes for delay was in government customers asking the wrong questions at the outset of wargaming projects,and AI is unlikely to help them identify the right questions and objectives for war

246、games faster.107 106 Author interview with think tank wargame design expert,11 January 2023.107 Author interview with private sector modelling and simulation expert,15 February 2023;Author interview with think tank wargame design expert,10 January 2023;Author interview with academic wargame design e

247、xpert,30 January 2023.Artificial Intelligence in Wargaming 39 6.Conclusions and Recommendations There are numerous ways in which AI may be integrated into analytic manual wargaming,but these use cases vary significantly in their technical complexity,practical feasibility,and level of operational ris

248、k.As such,this report has sought to distinguish between unrealistic use cases,and more realistic use cases where potential risks could be best managed,and where the benefits to UK decision superiority are significant enough to warrant further exploration.The AI use cases discussed throughout this st

249、udy(which are discussed along with their specific benefits and challenges in Appendix A)reflect a potential revolutionary change in wargaming,which is pointing to hybrid wargames and a potential bridge between the wargaming and modelling and simulation communities,which have historically come from c

250、ompeting epistemological views on the value of wargaming.It is perhaps in this breaking down of barriers between communities that senior decisionmakers will be able to inject more ingenuity of thinking and speed into decision-making,to contribute towards operational advantage.In conclusion,this stud

251、y found that AI has potential as a tool to supplement some low-risk elements of wargaming,but introducing AI wont overcome the delays and costs often associated with wargaming.Introducing AI in wargaming itself may in fact introduce new costs,administrative delays,and uncertainty in outputs.However,

252、AI may still contribute towards decision advantage by enabling new methods of data analytics for decisionmakers and enabling analysis of player decision-making,if some of the enabling AI techniques discussed in this report are advanced to become operationally viable.The following recommendations are

253、 focused on preparing the UK Defence community to develop and deploy effective AI-enabled wargaming systems in a responsible and evidence-based way.Recommendations 1.Invest in cross-cutting AI enablers The technical challenges identified in this study apply to various AI use cases,so cross-cutting e

254、nablers may be a more efficient use of resources to enable desirable applications.AI with contextual awareness,AI explainability techniques or AI that can establish causality between decisions and their effects would all help to unlock several AI use cases that could bring revolutionary benefits to

255、decision-making.Anna Knack and Rosamund Powell 40 2.Develop verification and validation(V&V)procedures for AI wargaming tools Experts highlighted the prevalence of non-AI automation being markets as AI,and risks around inexpert customers procuring low-quality AI-enabled wargaming tools.Standardised

256、V&V procedures will be important to ensure the reliability of AI tools for wargaming and their compatibility with rules of engagement,ethical standards and explainability requirements.3.Trial narrow AI use cases in parallel to investing in complex AI-enabled wargaming There is a high cost to benefit

257、 ratio in the more sensationalised AI use cases such as CoA analysis and adjudication explored in this report.There is merit in testing implementation of narrow and relatively low risk automation and AI use cases(e.g.Procedural Content Generation,first pass scenario generation,and speech transcripti

258、on)in wargames,while simultaneously investing in research to advance complex AI-enabled wargaming(e.g.AI for player decision-making analysis and AI in adjudication)in parallel.4.Commission research on wargaming epistemology and decision-making in wargaming A better understanding of an effective warg

259、ame and human decision-making during a wargame would enable the design of AI-enabled tools that augment rather than add uncertainty to the validity of wargame outputs.In strategic wargames,a better understanding of quantifiable parameters that affect strategic outcomes would be required.The values t

260、hat decision-makers apply in their decision-making would also need to be identified.41 41 Appendix A.Long-list of AI-enabled Techniques A longlist of AI-enabled techniques identified throughout this research project is provided below(as illustrated in Figure 5).For each use case,we describe the envi

261、saged benefits and challenges,drawing on insights across the research interviews,workshop,and literature review.Finally,we give an indication of whether workshop participants saw each application as meriting prioritisation.Views on this topic were highly variable and so in many cases divergent views

262、,alongside reasons for divergence,are given.These tables will provide a valuable reference point for anybody looking to learn more about a planned AI application for wargaming.Figure 5.Long-list of AI use cases for wargaming identified.Anna Knack and Rosamund Powell 42 A.1 Game design Table 5.Summar

263、y of AI applications for game design.Description of AI application Envisaged benefits Challenges Rationale for(de)prioritisation108 AI red team agents(e.g.via agent-based modelling)109 Reduces number of personnel required;Varied red team characteristics can be tested;Games can be easily repeated Red

264、uces ability to study human decision-making(becomes modelling and simulation);110 Training data frequently unavailable for realistic red-teaming;111 Emergent behaviours of AI agents are difficult to validate for realism;112 Actions of AI red teams are not explainable.113 Divergent views:Invest:Some

265、favoured investing in AI red teams for more tactical games.Track:Many participants thought AI had potential to be integrated into red team activities in the future but cautioned that current models are not sophisticated enough to capture complex human behaviour.Since there are several commercial ent

266、ities already working on this aspect,it will be better to track industry progress.Disregard:If the game itself is not digitised,it will be very challenging for AI to model the rules accurately.Additionally,modelling complex and unpredictable human interactions is not possible in the near term,especi

267、ally for less quantifiable,strategic games.108 Insights are drawn from a CETaS Workshop held on 16 March 2023.Where participants views were divergent,the order in which prioritisation rationales are presented herein is not indicative of which views were most popular.109 Author interview with academi

268、c wargame design expert,30 January 2023;Author interview with government AI in wargaming expert,26 January 2023;Author interview with government AI technical expert,19 January 2023.110 Author interview with government AI technical expert,17 January 2023.111 Author interview with government AI techni

269、cal expert,17 January 2023;Author interview with private sector modelling and simulation expert,15 February 2023.112 Author interview with academic wargame design expert,18 January 2023;Author interview with government AI technical expert,17 January 2023.113 Author interview with academic wargame de

270、sign expert,30 January 2023.Artificial Intelligence in Wargaming 43 Procedural Content Generation(e.g.,via deep learning)114 Increases speed of development of game mechanics;Improves game realism by adding in more detail to background content/non-player characters Lack of control of generated conten

271、t;115 High cost-to-reward ratio outside of commercial sector with big budgets Divergent views:Deploy:AI can be already do this within limited scope(e.g.producing certain wargaming rules,developing non-player characters).Invest:While the simplest forms of PCG could already be deployed,more investment

272、 is needed to harness its full potential.Track:Some thought it possible to piggyback off of developments in industry.Scenario and inject generation(e.g.,via generative AI)116 Increases speed of generation of text and visual aids(e.g.,DALL-E type tools)to the game to reduce time and costs;117 Improve

273、s game realism and player immersion Current techniques are only sophisticated enough for background content,or as a first pass which requires further edits by a human;118 third-party models are not always seen as sufficiently Divergent views:Deploy Several participants cited already using tools like

274、 ChatGPT to produce a first-pass version of scenarios,cutting significant time and costs to scenario development.This can free up time to focus on more critical game elements.Invest Participants mentioned that this type of input should be explored further by Defence to ensure that the creative eleme

275、nts of any content generation are robust.And,while a lot of content can be generated via simple automation already,investment in future advancements of this through AI were(broadly)seen as worthwhile to pursue.114 Jialin Liu et al.,“Deep Learning for Procedural Content Generation,”arXiv(October 2020

276、).115 Jialin Liu et al.,“Deep Learning for Procedural Content Generation,”arXiv(October 2020).116 Author interview with think tank wargame design expert,24 January 2023.117 Author interview with private sector modelling and simulation expert,1 February 2023.118 Author interview with academic wargame

277、 design expert,17 January 2023;Author interview with think tank modelling and simulation expert,2 February 2023;Author interview with private sector modelling and simulation expert,1 February 2023.Anna Knack and Rosamund Powell 44 trustworthy;119 loss of creative richness in in content.120 Game play

278、 testing(e.g.,via agent-based modelling,artificial neural networks and/or deep learning)121 Ensures a more comprehensive picture of possible outcomes of games;reduces risk of unexpected outcomes;helps identify unexpected mistakes in AI-generated content122 Expensive and time consuming for wargames t

279、aking place outside of simulated environments123 Divergent views:Deploy:Provided that human designers have the ultimate say in finalising different game play elements simulated by AI,several participants believed that models could help to test a wider number of potentialities in games compared to hu

280、mans.Track:Industry is ahead in this and there is a high risk that AI will not be able to model the game sufficiently to achieve this objective.Disregard:Other participants would disregard this use case for manual wargaming due to the high costs associated with creating a digital twin of the war gam

281、e to facilitate testing using AI agents.119 Author interview(2)with AI technical expert,11 January 2023.120 CETaS Workshop,16 March 2023.121 Author interview with government AI technical expert,19 January 2023.122 Daniel Karavolos,Antonios Liapis and Georgios Yannakakis,“Learning the patterns of bal

282、ance in a multi-player shooter game,”Proceedings of the 12th International Conference on the Foundations of Digital Games(August 2017):110.123 Author interview(2)with AI technical expert,11 January 2023.Artificial Intelligence in Wargaming 45 A.2 Game execution Table 6.Summary of AI applications for

283、 game execution.Description of AI application Envisaged benefits Challenges Rationale for(de)prioritisation124 Simulating opponents responses(e.g.,via predictive analytics)125 Reduces number of personnel required;novel opponent actions may be discovered Lack of data on real-life behaviour patterns o

284、f opponents;difficulties in verification and validation of outcomes;explainability challenges.Divergent views:Deploy:Some thought that AI could identify a wider range of possible opponent reactions,thereby identifying new directions in which the game could go.Invest:Further research could help eluci

285、date whether it is possible for AI to draw on historical outcomes to give probabilities on how opponents would react to specific situations.Warrants more exploration at tactical levels of warfighting and in deductive exercises because we have patterns of data,but less useful in strategic games.Track

286、:Several participants felt more research was needed on how accurate AI predictions were on this aspect of the game,since large margins of error here would risk skewing other game dynamics(e.g.,blue team responses).124 Insights are drawn from a CETaS Workshop held on 16 March 2023.Where participants

287、views were divergent,the order in which prioritisation rationales are presented herein is not indicative of which views were most popular.125 Author interview with government wargaming experts,10 December 2022.Anna Knack and Rosamund Powell 46 Adjudication(e.g.,via logistic regression)126 Makes adju

288、dication less random;allows integration of operational data or prior wargame data into future adjudication decisions Not sufficiently flexible to account for subjective nature of adjudication;127 lack of data;128 not sufficiently explainable129 Divergent views:Track:Scepticism about AI taking on thi

289、s role is widespread.Industry projects should be tracked to understand if this is something that could be considered later down the line,specifically in the context of tactical games given limited potential at the strategic level.Disregard:Sometimes,adjudication is based on rules and mechanics.AI do

290、es not have a place here as simpler automation is possible.Otherwise,adjudication requires complex human judgement and understanding of the strategic context,as well as the possible military tactics and strategies that may be deployed.These decisions are best taken by humans with expert knowledge in

291、 these fields.AI to run through repetitive tactical decisions(e.g.,via AI modelling and descriptive analytics)130 Increases player time available for more relevant/complex decisions.131 Cumulative impacts of small errors can build up to skew overall game outcomes.Divergent views:Deploy:Some suggeste

292、d this could already be deployed in narrow use cases.Track:Since many participants felt that AIs role here would not have too much impact on important decisions of the game(e.g.,at the strategic-level),this could be a great way of freeing up players from mundane considerations.Disregard:Others sugge

293、sted that decisions made over tactical aspects could in fact have cumulative impacts.As such,humans should still be responsible.126Jennifer McArdle and Eric Hilmer,“Effectively Integrating Technology into Wargames,”Interservice/Industry Training,Simulation,and Education Conference(2022).127 Author i

294、nterview with think tank wargame design expert,12 January 2023.128 Author interview with think tank wargame design expert,23 January 2023;Author interview with private sector modelling and simulation expert,1 February 2023.129 Author interview with think tank wargame design expert,24 January 2023.13

295、0 Author interview(2)with private sector modelling and simulation expert,1 February 2023.131 Author interview(2)with private sector modelling and simulation expert,1 February 2023.Artificial Intelligence in Wargaming 47 AI decision assistants and Course of Action(CoA)analysis tools(e.g.,via reinforc

296、ement learning)132 Increases speed of player decision-making;decreases stress of decision-making If the entire system is not digitised,it becomes difficult for AI to understand player moves and decisions.Divergent views:Invest:Defence research organisations are already investing here,and commercial

297、industry will not.Defence has to develop this capability in-house.Disregard:Technically infeasible because it is difficult for AI to make sense of an environment with no rules.Virtual assistants that provide reminders on legal and regulatory framework,rules of engagement,updates on real-world policy

298、 events and Reduces cognitive load on players;Improves ethical decision-making of players;improves player immersion.134 Slippery slope towards players playing a less active role in the game;135 lack of explainability contributes to distrust;136 likely to be a highly costly and technically complex de

299、velopment;137 establishing whose values,standards Divergent views:Invest:Given the decisions virtual assistants would be involved with,it is necessary to ensure they are tailored to the specific needs of defence stakeholders.Investment from within this sector is therefore needed.Participants see dev

300、eloping this capability as an enormous task,especially given the need to develop UK-specific tools for the UK legal context,but one with potentially significant benefits.It would be critical for any AI model in this capacity to start off with a more passive role supporting players(e.g.,pointing out

301、why they may have failed in winning a scenario)before gradually building up complexity.Doing so may also offer a useful training benefit where players can identify where they went wrong.132 Alec Barker,“BrainSTORM:exploring artificially intelligent COA development in STORM,”SPIE.Library(April 2021).

302、134 CETaS Workshop,16 March 2023.135 Author interview with academic wargame design expert,18 January 2023.136 CETaS Workshop,16 March 2023.137 CETaS Workshop,16 March 2023.Anna Knack and Rosamund Powell 48 news analysis(e.g.,via descriptive analytics and NLP)133 should be incorporated in ethical req

303、uirements will be challenging.Track:Currently,the level of complexity required for a role like this does not correlate to AI capabilities,meaning there is a serious risk that models could provide incorrect advice on important game mechanics(e.g.,legal rules).Participants felt this is something which

304、 could benefit from monitoring industry developments in other fields(e.g.,AI legal advisors),particularly since it is likely to be a highly costly and technically complex technique.133 CETaS Workshop,16 March 2023.Artificial Intelligence in Wargaming 49 A.3 Game analysis Table 7.Summary of AI applic

305、ations for game analysis.Description of AI application Envisaged benefits Challenges Rationale for(de)prioritisation138 AI to highlight interesting interactions between red and blue human agents to data scientists139 Reduces analysis time required;140 enables insights into tactical and strategic dec

306、ision-making that would enable decision advantage Limitations on AIs ability to identify what is salient mean important interactions may be missed;141 AI could misinterpret true emotions or illogical actions.142 Divergent views:Invest:A few participants advocating investing in this though further an

307、alysis was not provided.Track:Closely related fields such as diplomatic crisis management are making significant progress in this field and lessons can be learnt from them(e.g.,use of AI to track the convergence of opinions during peace negotiations).However,this still needs more development and ind

308、ustry was seen as being likely to be exploring this anyway so Defence should keep a close eye on developments.Data capture on player behaviour(e.g.,Enables analysis of player decision-making Wargaming becomes human subjects research when such invasive data Divergent views:138 Insights are drawn from

309、 a CETaS Workshop held on 16 March 2023.Where participants views were divergent,the order in which prioritisation rationales are presented herein is not indicative of which views were most popular.139 Author interview with government wargaming experts,10 December 2022;Author interview(2)with think t

310、ank wargame design expert,10 January 2023.140 Author interview with private sector modelling and simulation expert,15 February 2023.141Author interview with think tank wargame design expert,24 January 2023.142 Author interview(2)with private sector modelling and simulation expert,1 February 2023.Ann

311、a Knack and Rosamund Powell 50 via computer vision)143 collection is used,bringing further restrictions;144 infrastructure is often not there to facilitate sophisticated data collection;145 difficult to predict for a particular game what data are most important to capture;146 invasive data collectio

312、n protocols can impact the way players behave,introducing unexpected game outcomes.Deploy:Since this is an output that is challenging for humans to replicate manually,utilising AI for this role could open up new insights into player decision-making that would help to enrich other analytical processe

313、s.Track:Progress on computer vision is likely to be most advanced in other sectors and Defence can draw on this progress.Player decision-making Reduces possible bias in players self-reporting of decision May be less useful for operational analysis and more relevant to Divergent views:Deploy:While so

314、me participants placed this in deploy,they believed NLP can be used to support a human analyst,but the process certainly 143 Author interview with think tank modelling and simulation expert,17 January 2023.144 Author interview with think tank wargame design expert,24 January 2023.145 Author intervie

315、w(2)with think tank wargame design expert,10 January 2023.146 Author interview with private sector modelling and simulation expert,15 February 2023.Artificial Intelligence in Wargaming 51 analysis(e.g.,via NLP)147 logic;captures insights on why a move failed/succeeded quicker;generates insights on h

316、ow players make decisions under stress;suggests avenues for reducing bias or novel strategic or tactical approaches;creates database on player decision-making to one day emulate a player.148 academic studies on wargaming;149 lack of transparency over how AI arrives at a judgment on player-decisions

317、will likely create challenges in taking forward any AI-generated analysis.150 cant yet be fully automated.Thorough testing would also be required to check the accuracy of these systems in the defence context.Invest:Participants saw utility in applying AI for emotion detection and analysis of players

318、 behaviour to gather data and optimise decision-making performance under stressful conditions.Track:Many participants felt that there is not enough data or trials using AI solely for analysing human decisions in gaming,meaning it would need more R&D before considering integration.Additionally,there

319、was scepticism over how game analysts would be able to understand the logic chain or rationale behind why AI arrived at a judgement on decisions,which not only risks reducing trust,but reduces the richness of findings.147 Author interview with private sector modelling and simulation expert,24 Januar

320、y 2023;Author interview with government AI technical expert,19 January 2023.148 Author interview(2)with private sector modelling and simulation expert,1 February 2023.149 Author interview with think tank wargame design expert,23 January 2023.150 CETaS Workshop,16 March 2023.Anna Knack and Rosamund P

321、owell 52 A.4 Game logistics Table 8.Summary of AI applications for game logistics.Description of AI application Envisaged benefits Challenges Rationale for(de)prioritisation151 Real-time language translation(e.g.,via NLP)152 Enables international collaboration and understanding of non-English discou

322、rse during the game Challenges over AI capturing nuance,idioms,sarcasm or accents which could impact game decisions.153 Deploy:Seen as the easiest application to set up with significant value.However,there would be a need for thorough testing in particular with regard to whether these technologies c

323、an handle 1)highly specialised military terms and their different usage across countries,2)varied accents and 3)nuanced language use without introducing bias.Collating wargame data with secret and OSINT,automating risk analysis(e.g.,via Facilitates increased collaboration across departments;increase

324、s speed of decision-making;enhances realism of game and re-introduce complexity of real-world decision-making in Infrastructural and classification obstacles to this sort of data sharing;Divergent views:Deploy:Participants consider this collation to be technically possible now without any AI applica

325、tions.Track:Participants were concerned over how trustworthy existing AI applications are in securing this confidential data against cyberattacks,151 Insights are drawn from a CETaS Workshop held on 16 March 2023.Where participants views were divergent,the order in which prioritisation rationales ar

326、e presented herein is not indicative of which views were most popular.152 Author interview(2)with think tank wargame design expert,10 January 2023;Author interview with AI technical expert,11 January 2023.153 CETaS Workshop,16 March 2023.Artificial Intelligence in Wargaming 53 artificial neural netw

327、orks)154 game environment;strengthens evidence-base for critical decisions;could be applied to select customer requirements or questions to guide future wargames quicker.155 intrusions,etc.Nevertheless,they felt it would be useful to monitor any industry developments on this aspect.Speech transcript

328、ion(e.g.,via machine learning)156 Improves completeness of data capture;ensures transcripts are available to store and use in future game design/development/analysis;reduces potential note-taker bias Large quantity of data generated will often be of limited use and too time consuming to sift through

329、;concern over how AI could capture nuance and accents in different languages.157 Deploy:Simplest form of transcription technologies are seen as ready for deployment with thorough testing.Other adjacent technologies,such as summary technologies,would need further work to ensure they are explainable.R

330、isk of deployment is lower given the less technology interferes less with the players.154 Author interview with government AI technical expert,17 January 2023.155 Author interview(2)with private sector modelling and simulation expert,1 February 2023.156 Author interview with think tank wargame desig

331、n expert,24 January 2023;Author interview(2)with think tank wargame design expert,10 January 2023;Author interview with former government wargame design expert,18 January 2023;Author interview with think tank wargame design expert,11 January 2023.157 CETaS Workshop,16 March 2023.Anna Knack and Rosam

332、und Powell 54 Appendix B.Glossary B.1 Wargaming lexicon Wargame:For this study,we adopt the definition of wargaming used in the MoD Wargaming Handbook.Here,wargaming is defined as a scenario-based wargame model in which the outcome and sequence of events affect,and are affected by,the decisions made

333、 by the players.158 This player-centred definition allows us to explore the ways AI might alter the role of the players.However,wargaming is not a monolith.Many techniques and processes fall under this umbrella term or are closely related in some capacity.Further definitions are therefore provided below.The wargaming landscape:Table 9.Table displaying key wargaming definitions.Term Definition Anal