《CSET：人工智能和虛假信息運動的未來-Part 2（英文版）（90頁）.pdf》由會員分享，可在線閱讀，更多相關《CSET：人工智能和虛假信息運動的未來-Part 2（英文版）（90頁）.pdf（90頁珍藏版）》請在三個皮匠報告上搜索。

1、 December 2021 AI and the Future of Disinformation Campaigns Part 2:A Threat Model CSET Policy Brief AUTHORS Katerina Sedova Christine McNeill Aurora Johnson Aditi Joshi Ido Wulkan Center for Security and Emerging Technology|1 Executive Summary The age of information enabled the age of disinformatio

2、n.Powered by the speed and volume of the internet,disinformation has emerged as an instrument of strategic competition and domestic political warfare.It is used by both state and non-state actors to shape public opinion,sow chaos,and erode societal trust.Artificial intelligence(AI),specifically mach

3、ine learning(ML),is poised to amplify disinformation campaignsinfluence operations that involve covert efforts to intentionally spread false or misleading information.1 In this series,we offer a systematic examination of how AI/ML technologies could enhance these operations.Part 1 of the series desc

4、ribed the stages and common techniques of disinformation campaigns.2 In this paper,we examine how AI/ML technologies can enhance specific disinformation techniques and how these technologies may exacerbate current trends and shape future campaigns.Our findings show that the use of AI in disinformati

5、on campaigns is not only plausible but already underway.Powered by computing,ML algorithms excel at harnessing data and finding patterns that are difficult for humans to observe.The data-rich environment of modern online existence creates a terrain ideally suited for ML techniques to precisely targe

6、t individuals.Language generation capabilities and the tools that enable deepfakes are already capable of manufacturing viral disinformation at scale and empowering digital impersonation.The same technologies,paired with human operators,may soon enable social bots to mimic human online behavior and

7、to troll humans with precisely tailored messages.These risks may be exacerbated by several trends:the blurring lines between foreign and domestic influence operations,the outsourcing of these operations to private companies that provide influence as a service,and the conflict over distinguishing har

8、mful disinformation and protected speech.We conclude that a future of AI-powered campaigns is likely inevitable.However,this future might not be altogether disruptive if societies act now.Mitigating and countering disinformation is a Center for Security and Emerging Technology|2 whole-of-society eff

9、ort,where governments,technology platforms,AI researchers,the media,and individual information consumers each bear responsibility.Our key recommendations include:Develop technical mitigations to inhibit and detect ML-powered disinformation campaigns.Social media companies and Congress should inhibit

10、 access to user data by threat actors and their proxies.The U.S.government and the private sector should increase transparency through interoperable standards for detection,forensics,and digital provenance of synthetic media.Chatbots should be labeled so that humans know when they are engaging with

11、an AI system.Develop an early warning system for disinformation campaigns.Expand cooperation and intelligence sharing between the federal government,industry partners,state and local governments,and likeminded democratic nations to develop a common operational picture and detect the use of novel ML-

12、enabled techniques,enabling rapid response.Build a networked collective defense across platforms.Online platforms are in the best position to discover and report on known campaigns.Because these campaigns may occur across multiple platforms its important to share information quickly to enable coordi

13、nated responses.All platforms,regardless of size,should increase transparency and accountability by establishing policies and processes to discover,disrupt,and report on disinformation campaigns.Congress should remove impediments to sharing threat information while enabling counter-disinformation re

14、search.Platforms and researchers should formalize mechanisms for cross-platform collaboration and sharing threat information.Examine and deter the use of services that enable disinformation campaigns.As ML-enabled content generation tools proliferate,they will be adopted by influence-as-a-service en

15、tities,further increasing the scale of AI-generated political discourse.Congress should examine the current use of these tools by firms providing Center for Security and Emerging Technology|3 influence for hire.It should build norms to discourage their use by candidates for public office.Integrate t

16、hreat modeling and red-teaming processes to guard against abuse.Platforms and AI researchers should adapt cybersecurity best practices to disinformation operations,adopt them into the early stages of product design,and test potential mitigations prior to their release.Build and apply ethical princip

17、les for the publication of AI research that can fuel disinformation campaigns.The AI research community should assume that disinformation operators will misuse their openly released research.They should develop a publication risk framework to guard against the misuse of their research and recommend

18、mitigations.Establish a process for the media to report on disinformation without amplifying it.Traditional media organizations should use threat modeling to examine how the flow of information to them can be exploited by disinformation actors and build processes to guard against unwittingly amplify

19、ing disinformation campaigns.Reform recommender algorithms that have empowered current campaigns.Platforms should increase transparency and access to vetted researchers to audit and help understand how recommendation algorithms make decisions and can be manipulated by threat actors.They should inves

20、t in solutions to counter the creation of an information bubble effect that contributes to polarization.Raise awareness and build public resilience against ML-enabled disinformation.The U.S.government,social media platforms,state and local governments,and civil society should develop school and adul

21、t education programs and arm frequently targeted communities with tools to discern ML-enabled disinformation techniques.AI-enabled disinformation campaigns present a growing threat to the epistemic security of democratic societies.Our report focuses on the social media and online information environ

22、ment because they will be primarily impacted by AI-enabled disinformation Center for Security and Emerging Technology|4 operations.They are part of a larger challenge that has undermined societal trust in government and the information upon which democracies rely.While these recommendations may help

23、 stem the tide,the ultimate line of defense against automated disinformation is composed of discerning humans on the receiving end of the message.Efforts to help the public detect disinformation and the campaigns that spread it are critical to building resilience and undermining this threat.Center f

24、or Security and Emerging Technology|5 Table of Contents Executive Summary.1 Introduction.6 AI/ML Implications for Disinformation Campaigns.12 Enhance Surveilling the Information Environment.15 Boost Identifying Fissures with Sentiment Analysis.16 Enhance Segmenting Target Audiences.18 Build a Digita

25、l GAN Army.20 Scale Up Content Creation.23 Exploit Synthetic Image,Video,and Audio the Deepfake Problem.27 Enhance and Deploy Dynamic Bots.30 Boost Social Engineering of the Super-Spreaders.32 Exploit the Recommendation Algorithms.33 Boost Conspiracy Information Laundering.36 Automate Trolling.37 Sc

26、ale Up Mobilization with Personalized Disinformation.39 Key Findings.43 Recommendations.51 Conclusion.59 Authors.61 Acknowledgments.61 Endnotes.62 Center for Security and Emerging Technology|6 Introduction “The real problem of humanity is the following:we have paleolithic emotions,medieval instituti

27、ons,and god-like technology.”-Edward O.Wilson On February 22,2014,Ukrainians woke up to the news that their embattled president,Viktor Yanukovych,had fled to Russia.In the preceding months,Ukrainians had protested widespread corruption and Russias pervasive influence.Russia viewed the protest and Ya

28、nukovychs flight as a clear failure of its“Ukraine policy.”3 Five days later,Russian special forces,operating without military insignia,fanned out through the Crimean Peninsula.Assisted by a local militia that they had cultivated for years,these forces seized the Crimean parliament,television and ra

29、dio stations,and Ukrainian bases,perpetuating an illusion of a genuine rebellion.4 Cyber operations cut off the means of communication with Kyiv and blockaded local Ukrainian military units.5 In parallel with the physical operations,a sustained disinformation campaign bombarded the population of Cri

30、mea with narratives portraying the interim government in Kyiv as“fascists who threatened the Russian-speaking population with genocide.”6 The intense information onslaught on broadcast and social media was the beginning of a prolonged disinformation campaign effort to legitimize an occupation.7 Russ

31、ia took the world by surprise by exercising a“new way of warfare for the 21st century war.”8 It encompassed the use of military,cyber,and influence operations and exploited societal fissures to disguise an invasion under the veil of self-determination.A month later,Russian special forces did it agai

32、n.Masquerading as locals,they took over municipal administration buildings in Donetsk and Luhansk and manufactured a rebellion,launching a Russia-Ukraine war that would claim nearly 14,000 lives,and that continues today.These tactics were accompanied by yet another disinformation campaign that explo

33、ited,weaponized,and Center for Security and Emerging Technology|7 exacerbated societal polarization in Ukraine.In short,polarization became both a means and the ends of Russias disinformation campaigns.The age of information also enabled the age of disinformation.From its Cold War roots,disinformati

34、on has re-emerged as a tool of geopolitical strategic competition and domestic political warfare.9 Powered by the speed and scale of the internet and leveraging digital marketing techniques,disinformation operations,which we define as the covert,intentional spread of false or misleading information,

35、have weaponized social media platforms and fractured the information environment to sow discord and undermine trust.10 Their full impact is difficult to measure empirically,yet they have certainly increased social discord and proliferated widely.11 The evolution of disinformation operations is under

36、way.When Russia applied these tools to discredit domestic opposition and exploit fissures in democratic societies abroad,other nations took note.12 Chinese and Iranian disinformation operations,also honed against domestic audiences and regional neighbors,are now deployed far beyond their borders to

37、project power.13 Today,81 countries use social media to spread propaganda and disinformation,targeting foreign and domestic audiences through variety of automated and human-driven tactics.14 The techniques used in these campaigns highlight the technical,cognitive,and social nature of the disinformat

38、ion problem.They capitalize on the central features of todays social media platformsmaximizing user engagement and connection.They overwhelm cognitive resources under stress from the intensity of modern-day information environment,exploiting biases and heightening emotions at the expense of rational

39、 decision-making.15 They seek to deepen existing fissures within open societies,erode trust,and chip away at the shared values that form the common foundation critical to functioning democracies.Artificial intelligence and its machine learning(ML)techniques have the potential to deepen the threat po

40、sed by disinformation campaigns from adversary nations and non-state actors.China and Center for Security and Emerging Technology|8 Russia have stated the importance of winning the AI race and pledged investments ranging from hundreds of millions to billions of dollars in AI research and development

41、.16 The National Security Commission on Artificial Intelligence sounded the alarm about the potential of AI technologies to“increase the magnitude,precision,and persistence of adversarial information operations.”17 Concerns about AI-generated video and audio impersonations,known as“deepfakes,”have g

42、arnered significant attention from researchers and policymakers.18 However,there are other AI capabilities,such as generative language models,ML-enabled chatbots,and audience segmentation and sentiment analysis techniques that may be more impactful.The diffusion of generative language models and the

43、 commercialization of“AI-as-a-Service”presents a global challenge as societies grapple with the potential of AI-generated disinformation to threaten their epistemic security,decision-making,and trust.19 In this paper,we examine how advances in AI,specifically ML,are likely to augment disinformation

44、campaigns.We leverage the RICHDATA framework that we described in detail in our companion report,“AI and the Future of Disinformation Campaigns:The RICHDATA Framework.”20 This framework describes the building blocks and stages of disinformation campaigns from the perspective of those who build them.

45、Center for Security and Emerging Technology|9 Figure 1.RICHDATA Pyramid Source:CSET.We break disinformation campaigns into multiple stages.Through reconnaissance,operators surveil the environment and understand the audience they are trying to manipulate.They require infrastructuremessengers,believab

46、le personas,social media accounts,and groupsto carry out their narratives.A ceaseless flow of content,from posts and long-reads to photos,memes,and videos,is a must to ensure their messages seed,root,and grow.Once deployed into the stream of the internet,these units of disinformation are amplified b

47、y bots,platform algorithms,and social-engineering techniques to spread the campaigns narratives.But blasting disinformation is not always enough:broad impact comes from sustained engagement with unwitting users through trollingthe disinformation equivalent of hand-to-hand combat.In its final stage,a

48、n influence operation is actualized by changing minds or even mobilizing witting or unwitting victims to action to sow chaos.Each of these stages has associated techniques that are depicted in the figure below.Center for Security and Emerging Technology|10 Figure 2.RICHDATA Techniques Source:CSET.Ce

49、nter for Security and Emerging Technology|11 In this paper,we examine how advances in AI may enhance the stages and techniques of disinformation operations.We proceed in three parts.First,we identify the AI research areas most impactful for campaigns and how threat actors may apply them.Second,we an

50、alyze salient features of current campaigns and the implications of future AI applications.Our research findings confirm that these technologies have the potential to augment disinformation operators and exacerbate the disinformation crisis through scale.Specifically,ML natural language processing a

51、lgorithms are well suited to discerning emotion and sentiment from the data-rich online environment,opening up new avenues for social engineering.Large language models and the technology powering deepfakes,generative adversarial neural networks(GANs),provide a means to impersonate humans online and

52、create disinformation at scale.These technologies may enable social bots to blend in by better mimicking human behavior patterns and enhance AI chatbots to target humans with precisely tailored messages.These risks are exacerbated by several current trends:the blurring lines between foreign and dome

53、stic disinformation,the outsourcing of disinformation creation to companies that provide influence services,the open culture of AI research,and the potential dual-use nature of many AI capabilities.All of the above may converge to further erode trust and accelerate the spiral of cynicism which creat

54、es fertile ground for further seeds of disinformation to take root.Finally,we offer recommendations for how governments,social media platforms,media,and AI developers can prepare and respond to the malicious use of AI for disinformation campaigns.Notably,while our study examines applications of AI/M

55、L to disinformation operations from a threat actor perspective,this technology can likewise empower defensive solutions.An assessment of defensive applications to mitigate and counter disinformation is not within the scope of this report and is left for future studies.Center for Security and Emergin

56、g Technology|12 AI/ML Implications for Disinformation Campaigns Machine learningthe ability of machines to learn from complex and voluminous data,identify patterns,and make inferences about decision rulescan both complement traditional automated techniques already employed in disinformation campaign

57、s,and augment the campaigns in novel ways.21 The transformative contribution of ML to any field depends on the presence of three elements:data,algorithms,and computing power.22 Disinformation operations are no exception.Non-ML Automation for Data Collection Not all automated techniques rely on ML;ne

58、vertheless,they can be critical to collecting and harnessing the data that enables machines to learn.Malicious actors can collect relevant data by using open-source tools or by creating custom ones.For custom tools,they may use the application programming interface(API),a mechanism that many platfor

59、ms enable to allow other software to talk to and interact with them more efficiently through code.These tools enable key tasks such as crawling networks to scrape publicly available data,and formatting or conditioning the data for additional processing and analysis.There is a vibrant market of freel

60、y available,open-source data scraping tools to collect data from social media platforms and download them into an easy-to-process format,such as a spreadsheet.Data scrapers can often circumvent the platforms efforts to limit access to users posts.For example,Twint,a popular Twitter scraper available

61、 for free on GitHub,allows downloading of tweets,bypassing the limitations on quantity and rate of download,and the requirement to create a Twitter account imposed on developers that access this data through Twitter API.23 While they may not provide the breadth of data available through the API,scra

62、pers nevertheless allow malicious actors to sidestep platform vetting procedures and to access publicly facing user data,often used for nefarious purposes.24 Robust reconnaissance relies on harvesting and harnessing data about the target information environment.Ingesting vast volumes of data from di

63、verse sources is not a trivial exercise and may Center for Security and Emerging Technology|13 require dedicated software engineers and data scientists to build these systems in-house.First,building a central repository of dataa data lakerequires an environment that can collect data from different s

64、ources and in different formats.For example,data may come in simple spreadsheets of web links,databases of articles,archives of papers,or stores of images and video.25 Second,data needs a place to go,such as cloud storage from commercial cloud providers.Third,this data may need formatting before it

65、is ready for data analytics additional processing.This processsometimes called data wranglingcan be simple or complex,depending on sources of data.26 Alternatively,a well-resourced actor may procure commercially available solutions from niche companies who provide pre-processing services for“big dat

66、a”or end-to-end solutions from ingestion to insight.27 Once the data is ingested,sanitized,and curated,operatives can move on to finding insights,an area where machine learning techniques excel.Foundational Machine Learning Technologies The key ML technologies for disinformation campaigns include na

67、tural language processing and generation,and GANs.Natural language processing(NLP)and generation(NLG)allow ML systems to read,write,and interpret text.NLP combines computational linguistics,modeling of human language based on rules,and artificial neural networks.28 While earlier versions of NLP appl

68、ications were often rules-based systems explicitly programmed to perform specific tasks,todays NLP neural networks automatically extract,classify,and label elements of text,and assign statistical likelihood to each possible element.29 As of September 2021,the two most powerful language generation sy

69、stems are Generative Pre-Trained Transformer 3(GPT-3),built by OpenAI,and its Chinese language equivalent,the PanGu-Alpha,recently introduced by Huawei.30 These companies trained large language models with 175 billion and 200 billion parameters,respectively,on a trillion words of human text through

70、massive neural networks.31 These systems work with humans:operators pose questions,provide instructions,or prompt with examples of writing,to which the system responds with text in the Center for Security and Emerging Technology|14 same style.GPT-3 completes the text the operators provide by probabi

71、listically choosing each next word from a series of options based on the patterns it learned by analyzing human writing.32 Researchers are working with GPT-3 to create poetry,write op-eds,develop computer code,and even generate images.33 Developers are incorporating GPT-3 into apps to improve chat,t

72、ranslation,and ads.34 Diffusion of large language models is underway.A mere year since GPT-3 proved novel capability,more generative language modelssome of them larger and more complexhave been announced or released from U.S.,Korean,Russian,Israeli,and French companies,enhancing language generation

73、in more languages.35 Generative Adversarial Networks(GANs)use two neural networksa generator and a discriminatorto compete against one another.The generator attempts to create data to fool the discriminator.The discriminator seeks to detect fake data from the generator.As the neural networks compete

74、,they learn from each other,and the competitive process produces an increasingly realistic output in image,video,and audio formats.Disinformation actors can harness the latest AI/ML innovations cheaply by leveraging transfer learning,a technique that allows the knowledge gained by an ML model in one

75、 setting to apply to a different related setting.36 Well-funded AI researchers spend significant time and computing power training large neural networks on large datasets to solve a novel task.Researchers often release these pre-trained models into open-source code repositories so other researchers

76、can reproduce,learn,and build on the research.Anyone can download the released pretrained model and retrain on a smaller dataset to accomplish narrower tasks.This technique enables researchers to fine-tune algorithms to learn new tasks in a way that is faster and cheaper,and that uses less computing

77、 power and training data.37 Private firms specializing in disinformation for hire“influence as a service”are likely to augment their non-ML offerings with ML-enabled techniques.In a recent threat report,Facebook researchers highlighted the significant contribution to disinformation operations of pub

78、lic relations and digital marketing firms that specialize in Center for Security and Emerging Technology|15 online manipulation using deceptive means.38 Some firms already sell content generation services and amplification bots that follow,retweet,and like tweets,though it is unclear if these servic

79、es are ML-enabled.Many of them have hundreds of thousands of international customers,including celebrities,media pundits,candidates for public office,and state-run influence outlets,such as Chinas news agency Xinhua.39 Disinformation operators increasingly outsource some or all stages of building a

80、campaign to such firms,receiving plausible deniability in return.The AI research community can also be a source of new tools if they release their research without built-in mitigations against misuse.For example,well-resourced firms are likely to incorporate the fine-tuned ML systems into their menu

81、 of services and offer ML-enabled disinformation as a service.Disinformation actors are likely to draw on these services.These ML technologies form a powerful foundation of techniques that can enhance disinformation operators workflow.In the following sections,we discuss their demonstrated and poten

82、tial use for building disinformation campaigns.For each section below we highlight in blue the stages of the RICHDATA framework that are most impacted by these advances.Enhance Surveilling the Information Environment ML-powered predictive analytics capabilities can help threat actors identify curren

83、t and future social fissures.For example,economists at Google used the publicly available search trends databases“Google Trends”and“Google Correlate”to create an accurate prediction model for economic“nowcasting”predictions of the very near-future economic indicators.40 The same methodology could th

84、eoretically be leveraged with different data points from these databases to create an engine that can accurately understand and predict near-future social trends in specific geographic regions,though this is speculative.Center for Security and Emerging Technology|16 Virality,the tendency of content

85、or topics to circulate rapidly and widely online,is a key goal for threat actors to achieve maximum engagement.41 Researchers have used ML techniques to accurately predict which news events are likely to go viral early on in the contagion process.42 The ability to predict the virality of organic con

86、tent can confer an information advantage to threat actors because it gives them more time to spin narratives in their favor and exploit organic content more effectively.It may also give them insights into which narratives are more compelling and may deserve greater resources.Predicting message viral

87、ity is relevant in both initial reconnaissance and for amplification.Advances in ML are poised to make tools for social listeningmonitoring social media mentions of a topicmore powerful and more precise.Commercial social listening tools combine network crawling and social media monitoring for engage

88、ment and mentions,two key indicators of audience interest,with analysis of users sentiments.43 Some tools do not use ML,but simply allow a user to build a keyword search string and receive an alert when this term is encountered on the web,not including social media.44 More sophisticated tools claim

89、to listen to millions of conversations across social media platforms and news sites,gauge the sentiment of the audience,analyze images,and produce comprehensive analytics of conversations.45 Some commercially available tools provide a glimpse into how ML is enabling powerful social listening with ad

90、vanced multilingual text analysis and visual analytics.46 Threat actors can use similar technologies to better understand the target information environment and to provide feedback throughout their campaigns.While these and similar tools have limitations,they will continue to evolve and improve with

91、 the application of ML capabilities.Boost Identifying Fissures with Sentiment Analysis Sentiment analysis,a research field within NLP,extracts subjective qualitiesattitudes and emotions such as sarcasm,confusion,and suspicionfrom text and interprets the text as negative,positive,or neutral.Some ML m

92、odels can determine the degree of Center for Security and Emerging Technology|17 polarization from user comments and posts with 94-96 percent accuracy.47 In one study,NLP and text classification techniques were used to pick up on linguistic patterns and identify depression in Reddit users.48 While s

93、ome skepticism persists about the efficacy of its early versions,sentiment analysis continues to improve.More sophisticated ML-enabled tools trained on datasets curated by human linguists are better able to grasp the colloquial language,abbreviations,emoticons,emojis,and slang used on social media.S

94、entiment analysis tools allow tech-savvy disinformation actors to identify targets who post supportively or critically about certain topics.The current heuristics for doing so depend on inferring information,such as political leanings,from other contextual data,such as groups users belong to and peo

95、ple they follow.Sentiment analysis could allow disinformation operators to bypass this and identify a users leanings based on their posts.State-of-the-art offerings can extract sentiment from text in over 13 languages and deploy the capability on the disinformation operators network.49 While these c

96、ommercial solutions can be expensive,new tools are appearing on the underground market and drawing on openly released research.50 Sentiment analysis could also help determine the alignment of groups or individuals based on characteristics such as race,gender,or tone.Threat actors could build persona

97、s that are likely to resonate with an individual or group,feeding into their existing beliefs and precisely targeting vulnerable audiences to drive up engagement and influence behavior.Stance detection takes sentiment analysis a step further,using ML techniques to classify text against a predefined

98、concept.51 Stance detection algorithms can identify not only whether the sentiment of a post is negative or positive,but whether it agrees with a broader ideasuch as atheism or feminism.52 Operators could use this to identify supporters and their communities during the reconnaissance phase of an ope

99、ration.While the use of these algorithms currently requires an understanding of supervised ML in addition to basic web scraping,they are likely to be integrated soon into publicly available tools.Center for Security and Emerging Technology|18 Researchers have adapted and tested a stance detection sy

100、stem across English,French,Italian,Spanish,and Catalan,including exploring the content of a tweet and deciphering contextual information.53 Stance detection enables more accurate targeting of individuals by providing a deeper understanding of a users positions and the strength of their beliefs.Such

101、targeting can be further refined by combining this information with user data available from data brokers and a users public posts.Enhance Segmenting Target Audiences Similar to social listening,social network analysis tools help visualize social and knowledge networks.Some tools use network graph t

102、heory,statistical analysis,and visualization techniques to show how nodespeople or organizationsare linked together,and the nature of their relationships.54 Others apply natural language capabilities to process unstructured data and identify the targets,groups,and key connections between them.55 The

103、se or similar tools allow operators to identify connections within and between groups,and key players to target.ML techniques can help infer the attributes of users by examining salient aspects of their network.In a 2019 study on the popular Chinese social media site Renren,researchers used availabl

104、e data,such as a users friends list,to infer unstated information,such as their age,hobbies,and university majors.56 This network analysis technique allows malicious actors to make predictions about groups of users to infer strengths of relationships,similarity in interests,and potential susceptibil

105、ity to influence on topics based on the posts of those in ones network.Threat actors may use tools provided by the platforms to augment their audience segmentation efforts.The ever-increasing demand for precision advertising drives platforms to give third party applications and organizations access

106、to nuanced audience segmentation tools.Using available APIs,applications can interact with the platform in real time,collect data,analyze it,and post content.Some products use this legitimately acquired data along Center for Security and Emerging Technology|19 with ML to predict consumer trends.57 O

107、ther commercially available tools purport to use unsupervised ML techniques to segment audiences,tease out sentiment,and map personas,producing visualizations of consumer attitudes.58 However,operators can use these same tools for malicious purposes.Platforms connect advertisers with their target au

108、diences of digital“look-alikes.”ML algorithms can recognize when users belong to a particular target audience and when their pattern of behavior on the platformsuch as the type of content they engage withmatches that of other users.Some platforms provide built-in ML-enabled services to help advertis

109、ers and political consulting firms find new audiences through matching techniques.59 They collect data points about each of their users,including data from other websites they visit,apps on their phones,and location data,although privacy-savvy users can opt out of this collection.60 If threat actors

110、 are able to masquerade as legitimate advertisersparticularly by hiring firms that provide influence servicesthey too can find new look-alike audiences to target based on users past pattern of online behavior on and off the social media platforms.Psychographics is an advanced form of demographics th

111、at includes profiling of values,interests,activities,and opinions of population segments for advertising and election targeting.61 Proprietary technology from firms like now-defunct Cambridge Analytica built psychographic profiles of people on the basis of the Big Five Model of personality traits:op

112、enness,conscientiousness,extraversion,agreeableness,and neuroticism.62 While Cambridge Analytica harvested user data in violation of Facebooks terms of use and its claims of efficacy are disputed,there are other,legitimate ways to collect psychographic information,including surveys,questionnaires,qu

113、izzes,website analytics,browsing data,psycholinguistic dictionaries,and social media engagement(likes,clicks,posts).Some commercial solutions combine psychographics,sentiment analysis,and deep learning to extract insight from text,including sentiment,emotion,and personality traits.63 Combined with d

114、emographic and“pattern of life”data,psychographic profiling and similar publicly available tools can help Center for Security and Emerging Technology|20 threat actors target their messages precisely,provided they have enough user data.Between voter files,political data firms,commercial data brokers,

115、and social media platform data,candidates for public office can have access to over three thousand data points on every voter.64 Some threat actors may also have access to this data.More than 3.5 billion users saw their personally identifiable information,emails,photos,and credit card data stolen in

116、 the top breaches of the twenty-first century,including biometric fingerprint data of 5.6 million U.S.government employees with security clearances.65 Build a Digital GAN Army As GANsthe ML capability behind deepfakessupercharge the production of believable images of nonexistent humans,threat actors

117、 have taken notice.66 The use of ML-generated profile pictures by disinformation campaigns is accelerating.One website,illustrates this phenomenon.With every refresh of the site,a new fake face emerges,with an image of exceptional quality whose inauthenticity is practically undetectable with the hum

118、an eye.Days after the website was created,it came to the attention of malicious actors,who promptly incorporated the new face generation capability into their operations.Just weeks later,Facebook discovered high-quality ML-generated profile avatars in a disinformation campaign on the platform.67 Fac

119、ebook quickly banned the use of GAN-generated content and relied on other markers,such as coordination,to detect the campaign.GAN-generated photos remain difficult to detect in real time,as detection and generation technologies continue to leapfrog one another.Center for Security and Emerging Techno

120、logy|21 Diffusion of GAN-generated Avatars In just one year,the use of GANs went from a rarity to a staple of disinformation campaigns.In June 2019,in the first publicly known case,Katie Jones,an inauthentic persona on LinkedIn with a GAN-generated avatar,connected with scholars and high-profile fig

121、ures in the national security community in a likely espionage operation.68 By December 2019,Facebook had removed an inauthentic network with GAN-generated avatars operating groups for a content provider connected with the Epoch Media Group,a hyper-partisan U.S.media outlet with a special emphasis on

122、 coverage of China.69 In August of 2020,the“Spamouflage Dragon”operation deployed a network of GAN-generated avatars on Twitter and YouTube posting in support of the Chinese Communist Party and targeting U.S.audiences with messages critical of the U.S.response to the COVID-19 pandemic and its China

123、policy.70 Later in August,in a“Peace.Data”operation,a network of Russian Internet Research Agency(IRA)-linked personas with GAN-generated avatars on Facebook,Twitter,and LinkedIn,targeted left-wing audiences in the United States and the United Kingdom.Masquerading as editors for a media site,they hi

124、red unwitting freelance writers in both countries to write stories aimed at steering progressive voters away from moderate candidates.71 By October 2020,a parallel“NAEBC”IRA-linked operation used a network of GAN-generated avatars to target right-wing audiences on mainstream platforms as well as Par

125、ler and GAB.72 By early 2021,fake personas with GAN-generated avatars engaged in rhetorical combat in information environments across the Central African Republic,Mali,and Libya,and attacked the Belgian governments plans to limit access of“high-risk”suppliers to its 5G network.73 The barriers to acc

126、essing GAN-generated avatars are low.A novice operator can navigate to a site like and save the generated photo,yielding a unique profile picture.An actor with limited coding skills can automate navigating to the site Center for Security and Emerging Technology|22 and scraping photos,yielding a libr

127、ary of untraceable avatars faster and at scale.Using these readily available tools,an operator can easily and cheaply build a campaign around a set of randomly generated avatars.Advanced actors seeking to target a specific racial or ethnic group can get a tailored set of avatars with some ML effort.

128、They can finetune pre-trained models for face generation such as StyleGAN2 on a targeted dataset of photos of individuals or groups.At the 2020 BlackHat cybersecurity conference,threat intelligence researchers demonstrated exactly this,warning about how easily threat actors can leverage publicly ava

129、ilable ML models for deception.74 Using a dataset of publicly available photos of Tom Hanks,they finetuned StyleGAN2 to generate new fake photos of the actor.If threat actors want to create a campaign targeting seniors in retirement homes or military veterans,they can assemble a dataset of publicly

130、available photo samples,finetune a model on this data,and get custom versions of thisseniordoesnotexist or thisveterandoesnotexist.Armed with avatars,they can build a supply of nonexistent humans for a tailored campaign targeting a specific group.In addition,generating faces to precise specification

131、s on demand may soon be part of publicly available applications.Researchers have already built an AttentionalGAN model,capable of generating images from a text prompt.This technology is rapidly improving and may soon be able to generate a photo of a fake human from a textual descriptiona photo of a

132、nonexistent person fitting specified parameters,generated on demand.75 In another effort,developers have combined the GPT-3 language generation system with a StyleGAN2 application to do just that.A text prompt asks GPT-3 to produce“a white female with blonde hair and green eyes.”Eight photos of none

133、xistent humans fitting the description emerge.76 The developers of this experiment plan to release this software in a public beta,for marketing professionals to generate faces for advertising purposes.It is unclear if or how the creators plan to guard against its misuse.Access to such an application

134、 will make it trivial for disinformation operators to order human faces of a specific race and ethnicity,outfitting digital troops to act as credible messengers in a campaign targeting ethnic minorities.Center for Security and Emerging Technology|23 Beyond avatars,fine-tuning a pre-trained model can

135、 be useful in other aspects of creating inauthentic persona profiles.The persuasiveness and longevity of a fake persona depends on an authentic look,an identity with presence across multiple social media platforms,and a credible back story.77 Here too,a series of fine-tuning projects have created a

136、pipeline of free images to build out the rest of an inauthentic profile.For example,leveraging openly available demos,threat actors can build a profile of someone who has a fake cat,posts photos of fake food,lives in a fake apartment,appreciates fake art,and has a fake LinkedIn resume working for a

137、fake start-up.78 Researchers and developers have begun cataloguing ongoing experiments with pre-trained models.*At best,these efforts offer a“go to”catalogue for malicious actors to manufacture fake lives.At worst,they demonstrate how disinformation operators or third-party“influence-as-a-service”fi

138、rms can build custom semi-automated capabilities.79 Scale Up Content Creation ML may be the biggest game-changer in the content creation stage of disinformation operations,with the potential to increase the scale and virality of campaigns.Recent breakthroughs in NLP coupled with synthetically genera

139、ted text,memes,video,and audio could make content richer and more compelling.Such tools will increase content creation and automate substantial parts of this human labor-intensive process as part of the human-machine team.80 This is not limited to text generation,as there are also prototypes to crea

140、te realistic,narrated videos and images from text prompts.81 ML-powered language translation and autocomplete functions can help foreign disinformation actors sound authentic,eliminating awkward phrasing and grammatical mistakes.Here,even relatively basic software can pay dividends.Google Docs has i

141、ntegrated ML into its Smart Compose feature,identifying grammatical errors and *See a catalogue of such experiments on http:/.Center for Security and Emerging Technology|24 autocompleting sentences in real time.82 Hijacking authentic posts or hiring native English speakers to write content,as has oc

142、curred in previous campaigns,is noisy and increases the potential for human leaks or exposure,whereas predictive composition is fast and cheap.83 Large generative models offer threat actors an“autocomplete on steroids”capability.84 In a disinformation content farm,large language generation systems l

143、ike PanGu-Alpha or the openly available GPT-J could streamline the labor-intensive process of creating text content.85 A few human operators working with a language generation system can create pipelines of blogs,short posts,memes,and“junk news”from a few prompts.These systems do not have to fully r

144、eplace humans to be powerful additions.They can do the heavy lifting,producing posts with prescribed number of characters,keywords,and themes of the day.86 These systems can also lessen the cognitive load of human operators such as those employed in a content generation“farm”by the Russian IRA.87 Th

145、is could reduce the problem of overworked human operators reusing talking points,lines,and phrases because they have to meet demanding content production quotas over long shifts.88 Future campaigns may be able to significantly scale down the number of human content creators by using language generat

146、ion models,and minutes later get back an array of content tailored to an audiences political preferences and biases.These systems can allow humans to shape the output and experiment.Such systems never tire nor suffer from poor morale or a crisis of conscience.The quality of disinformation narratives

147、 created by systems like GPT-3 is largely untested,but the machines ability to replicate the style and viewpoints of prompts shows promise.With just a few samples of extremist content,researchers have demonstrated its ability to mimic the style of far-right extremist writing and produce manifestos f

148、rom multiple viewpoints.89 A recent CSET study indicates that GPT-3 can craft stories that fit a viewpoint,manipulate narratives with a slant,seed new conspiracy narratives,and draft divisive posts that exploit political wedges and advance Center for Security and Emerging Technology|25 racial and et

149、hnic stereotypes.90 Experiments also show that its content is persuasive and appears authentic.For example,in one study,after seeing five tweet-sized messages written by GPT-3 and curated by humans,the percentage of survey responders opposed to sanctions on China doubled.In another study,73 percent

150、of human responders judged stories generated by GTP-3s predecessor model as credible and mostly indistinguishable from those written by journalists,particularly as the partisanship of content increased.91 Skeptics argue that GPT-3 often gets the facts wrong and loses the logical structure of its arg

151、uments after a few paragraphs.However,inaccuracy and inconsistency are not necessarily impediments for disinformation campaigns,especially with human oversight to curate the machines outputs.More troubling yet is the impact that synthetic text generation systems may have on(Hack)-Forge-Leak operatio

152、ns.92 Because generative language systems can mimic a style of writing from just a few samples,these capabilities could enable targeted and precise content production.Previous disinformation operations illustrated how threat actors have sought to weaponize forgeries of official correspondence to sow

153、 discord among U.S.allies.93 Threat actors seeking to impersonate and discredit specific public figures or organizations could conceivably do so at scale by creating believable forgeries,infusing them into a payload of hacked emails,and releasing them.Researchers have already demonstrated GPT-3s abi

154、lity to generate emails from bullet points or re-write them to sound more polite.94 Malicious actors could post forged emails,looking stylistically authentic and without any metadata that could provide forensic insight,which could be leaked to a media outlet in the closing weeks of an election campa

155、ign,for example.Yet with no technical way to detect machine-generated text,media outlets would face the task of verifying the unverifiable.For a threat actor,there may not even be a need to hack.They can prompt a language model with a few public samples of a public figures writing,forge thousands of

156、 believable emails or messages,and claim that the emails came from a hack,giving the dump the illusion of authenticity.Thus far,there have been no publicly known cases of threat actors operationalizing large language models for disinformation Center for Security and Emerging Technology|26 campaigns.

157、There are also no effective methods to detect their use.To social media platforms and internet browsers,synthetic text looks like any text on the internet.This means that if these systems were deployed today,the platforms would likely not be able to distinguish between human and machine-generated te

158、xt.Early on in its fielding,GPT-3s creators at OpenAI were concerned with the misuse of this technology.They decided to control and vet access to the systema somewhat controversial decisionto enable innovation while guarding against misuse.But the proof of concept is out there.What sets GPT-3 apart

159、for now is the size of its training data and its neural network but the gap is closing fast.While recreating a model like this from scratch is expensive,these are no barriers for advanced actors.This is evidenced by Chinas Huawei announcing the PanGu-Alpha,a predominantly Chinese-language model,Russ

160、ias Sberbank producing a smaller version of GPT-3 Russian-language model,and South Koreas Naver releasing a Korean-language HyperClova generative language model.95 The system also does not need to be as advanced to be effective.The chances of diffusion increase as GPT-3 is commercialized,as is the l

161、ikelihood that large language models will continue appearing in the open source or offered on subscription basis as a service.96 EleutherAI,a collective of AI researchers advocating for open publication of language models,openly released GPT-J,a 6 billion parameter model,and plans to release larger

162、replicas of GPT-3.Israeli firm AI21 recently announced a public beta for its Jurassic 1 Jumbo system,a 178 billion parameter generative language model,and plans to provide access to its capabilities to wide array of organizations and individuals for a fee.97 As open-source and subscription-based lan

163、guage models emerge,well-resourced disinformation actors are likely to take advantage of them to saturate the information environment with believable falsehoods.Rapid innovation is also supercharging production of engaging visual content,potentially making the work of disinformation operators easier

164、 and faster.For example,Chinese tech giant Baidu created a prototype that generates more than one thousand narrated video summaries of news stories daily.Powered by computer vision and NLP,the system VidPressbuilds two-Center for Security and Emerging Technology|27 minute videos in less time than it

165、 would take humans.It reads an article,synthesizes key ideas,weaves them into a script,converts text to audio,collects related images and video clips,and combines them into a short video to deliver the content in a matter of minutes.98 While this prototype was built for news curation,it has direct a

166、pplications for disinformation campaigns.Text-to-Image conversion technology may soon enable the creation of images from a text description.99 DALL-E,a neural network trained to create images from text captions expressible in natural language is the newest innovation from the OpenAI Lab.100 DALL-E i

167、s a 12 billion parameter version of its sister model,GPT-3,and is able to generate images from text descriptions,including animals,objects,historic relics,and cityscapes.While its creations still lack believability,this area of research is rapidly growing.Researchers from the Beijing Academy of Arti

168、ficial Intelligence recently unveiled Wu Dao 2.0,a 1.75 trillion parameter multimodal system that can generate multiple types of digital media such as images and text from a prompt to illustrate a concept.101 Once the messages of the campaign are clear,images may eventually be created on demand to s

169、upport the narrative.That said,it remains to be seen how well this technology can compare to the current visual disinformation,such as cartoons,caricatures,memes,and photoshopping adversary heads of state into unrelated contexts.Even the province of humorous meme creation is no longer strictly human

170、-only.Humans are still better at making the memes funny,but how long this will remain the case is unknown.102 While a predecessor to GTP-3 trained on a dataset of jokes could mimic the written shape and style of some jokes,most lacked the quality of true humor.103 However,humor is not a prerequisite

171、 for a meme to go viraldivisive and hateful memes spread too.Exploit Synthetic Image,Video,and Audio the Deepfake Problem Deepfakes are a product of GANs,the same deep learning concept as the one used for generating synthetic avatars.In recent years,Center for Security and Emerging Technology|28 the

172、 number of deepfakes circulating on the internet has grown exponentially.Their uses include the spreading of nonconsensual,fake intimate imagery,the impersonation of employees to defraud a company,and the taunting of opponents in political campaigns.104 While highly anticipated,no deepfake videos ap

173、peared as part of foreign influence campaigns in the 2020 U.S.elections.However,the FBI continues to warn about the increased risk of their use in cyber and foreign influence operations.105 Making deepfakes is getting easier,and access to large quantities of data is no longer a barrier.In May 2019,r

174、esearchers at the Samsung AI Lab in Moscow built animated,talking head videos of Albert Einstein and Leonardo da Vincis Mona Lisa using a single image as input.106 This was a significant breakthrough given that previous examples had required a large dataset of images of the specific individual.107 H

175、ead-swapping,or juxtaposing an individuals head onto someone elses body,is similarly getting easier,although it still requires an actor or a body double to perform body actions and impersonate mannerisms.108 Similar advances are taking place in synthetic audio.109 Some researchers are skeptical of t

176、he deepfake hype and suggest that their weaponization will depend on pragmatic considerations,such as ease of access,decreased cost,and diminishing effectiveness of current techniques that are not ML-enabled.They argue that as long as less automated techniques produce content that can go viral,opera

177、tors have less incentive to invest in deepfakes.110 Well-resourced operators could plausibly integrate deepfakes into their arsenal,but decisions to deploy may depend on risk tolerancethe risk of being discovered weighed against an unclear reward.Defensive forensic capabilities increasingly focus on

178、 quickly detecting deepfakes of national leaders and high-profile public figures,which can further deter potential deployment.Nevertheless,deepfakes may be incorporated in disinformation operations in several plausible scenarios.Deepfake video and audio may become more prevalent on newer platforms,w

179、here engaging content can go viral quickly and there is limited verification that speakers are who they claim to be.For example,a person claiming to be Brad Pitt drew thousands into a Center for Security and Emerging Technology|29 conversation on Clubhouse,before he was identified when someone becam

180、e suspicious of his voice.111 Researchers at Mandiant already demonstrated how fine-tuning a publicly released model from a few voice samples can produce a cell-phone quality audio of Tom Hanks saying what researchers prescribed him to say.112 Malicious operators using samples of less recognizable v

181、oices than world-renowned actors,such as a military leader,a member of Congress,or another government official,could plausibly create audio impersonations that amplify disinformation.Disinformation operators could target newer platforms that are optimizing growth over security to test new techniques

182、 and fine-tune their efforts.Threat actors may also integrate tailored deepfakes into a hack-forge-leak type of disinformation campaign.A“leaked”deepfake video of U.S.or allied troops committing atrocities,for instance,has long been discussed as a possible catalyst for unrest or international crisis

183、.Disinformation operators could deploy deepfakes against diplomats to sow discord among allies.113 In a plausible election disinformation operation,a tailored deepfake of a political candidate or their family member delivering extreme comments could surface online as part of an alleged hack and is t

184、hen“leaked”in the weeks leading up to an election.Funneling this material through domestic political actors would further allow threat actors to exploit the heightened partisanship of the election cycle and the limitations in the ability of U.S.authorities to investigate constitutionally protected s

185、peech by domestic actors.Such a disinformation campaign could potentially sway undecided voters in key swing states.Even debunked,this type of content is engaging enough to go viral,exposing millions to disinformation before it is detected and throttled by social media platforms.Finally,and most imp

186、actfully,the slow drip and normalization of deepfakes may lead to an overall erosion of trust.In recent years,the number of deepfake videos has grown exponentially,doubling every six months.114 While a disinformation campaign using deepfakes would likely be discovered eventually,their growing popula

187、rity may erode trust in video evidence of wrongdoing.This“liars dividend”effectthe dismissal of a truthful occurrence as a deepfakemay further erode societal trust in government Center for Security and Emerging Technology|30 institutions and democratic process.115 In a polarized society,the prolifer

188、ation of deepfakes may pose a pernicious challenge.The long-term erosion of trust and inability to separate fact from fiction at a societal level can exacerbate the cycle of cynicism,creating fertile ground and increased vulnerability to disinformation,with or without deepfakes.Enhance and Deploy Dy

189、namic Bots As the social media platforms improve their detection capabilities,the life spans of inauthentic digital armies and their windows to blast disinformation are getting smaller.But ML can help the next generation of social bots become better at mimicking human behavior and achieve maximum am

190、plification in shorter time frames,augmented with ML-generated avatars.116 For example,researchers proposed to enhance social bots by extracting features from text and image content and processing them through neural networks to allow a social bot to publish contextually relevant comments.117 While

191、research literature theorizes the evolution of social bots with ML-enabled capabilities,open-source repositories and dark markets so far offer only limited end-to-end solutions.118 Meanwhile,AI developers were among the early innovators to use social media platforms as a testing ground for sophistic

192、ated bots.119 The techniques of bot detection services can be exploited to help build bots that behave more like humans.Supervised learning techniques use data curated by humans to teach the machine what is a“human”and what is a“bot.”Threat actors can reuse the training datasets released by bot dete

193、ction services,which examine features of a profile,number of followers,time of activity,language,and sentiment to classify a Twitter account as“likely bot”and“likely human.”120 In addition,unsupervised ML can help build more human-like bots.In theory,a model trained through unsupervised learning tec

194、hniques to draw insights from data could distinguish between human and automated patterns in how accounts connect with each other and spread information.This can help threat actors develop Center for Security and Emerging Technology|31 social bots that better imitate the human pattern.121 In an atte

195、mpt to improve bot detection systems and test their limits,researchers are exploring these approaches to build bots that would beat the bot detection systems and get classified as human.122 They aim to point out weaknesses in existing bot detection systems and improve them.Yet this line of research

196、could also show threat actors how to build bots that are indistinguishable from human-operated accounts online.Some sophisticated bots can already emulate human patterns of posting and consuming content.123 It is an open question,however,whether this will achieve the same level of virality as curren

197、t computational propaganda techniques.Bot operators may lose reach as they try to emulate humans,unless they create content that is engaging enough to go viral organically.ML techniques may soon enable social bots to maximize amplification and evade detection with varied synthetic content.Simpler ve

198、rsions of social bots tended to post pre-scripted set of phrases and keywords programmed into their code with a pattern of repetition that can get flagged by platforms.If synthetic text models can increase the variety of terms and phrases to resemble normal human speech,the bots may fool automated d

199、efensive systems.Social bots may not replace humans altogether.Rather,an advanced social bot system could combine already-available building blocks into a human-machine team or an entirely automated solution.Human operators can theoretically team a social bot with social listening tools and ML gener

200、ative language systems.124 First,this system could detect trending topics and hashtags through social listening tools and decipher the sentiment with a sentiment analyzer.Then,human operators could instruct the language generation component with prompts and examples to generate posts optimized aroun

201、d the trending subjects,a target audiences identity,and the disinformation campaigns narratives.This system could generate a pool of varied messages that appear authentic and relevant to the real-time online conversation.Next,a human could curate the most relevant outputs and feed these into Center

202、for Security and Emerging Technology|32 a pool of messages.Finally,the social bot can automatically pull messages from the pool and post them.Threat actors could plausibly develop such a system and,with some engineering ingenuity,automate human curation out of the loop entirely.Given the current mea

203、ns by which most ML language generation systems workthrough promptsfully automating an advanced social bot that will dynamically generate its own tweets and hijack trending topics in real time may be difficult.However,it will become more probable as more developers make their language models freely

204、accessible.If threat actors do not care about the quality of the output and are willing to risk the social bot occasionally posting nonsensical statements,they could risk automating the social bot end-to-end.Finally,more plausibly,the next steps in social bot evolution to mimic human behavior may co

205、me from the capability to respond dynamically and interactively as social bots integrate with chatbots,an AI-enabled technology we examine in a later section.125 The above scenarios assume that the same type of API access that the platforms expose to“the good bots”for legitimate uses is also availab

206、le to bots that are able to mask their true nature from the platforms vetting process.Boost Social Engineering of the Super-Spreaders ML systems may enhance social engineering techniques for targeting influencers,or“super-spreaders,”who can unwittingly amplify the campaigns messages.Threat actors ha

207、ve already successfully impersonated experts using inauthentic personas and duped news outlets with large followings.126 By assembling a dataset of posts scraped from accounts of public figures or acquired from the platforms through API access,threat actors could apply sentiment analysis and stance

208、detection techniques to identify frequently discussed topics that super-spreaders reshare.They could then generate synthetic content similar to collected posts to entice the super-spreader into unwittingly resharing the new content.Center for Security and Emerging Technology|33 While some of this is

209、 speculative,all the building blocks are available today.The amount of engineering effort may seem prohibitive to a small disinformation outfit or overkill to get a specific influencer to retweet a meme.But for an advanced state-sponsored threat actor,this is a long-term investment.Multiple high-pro

210、file figures can be cultivated with ML-enabled tools,assuming they manage their social media presence themselves.A series of posts from a super-spreader can reach millions of followers,receive additional amplification from broadcast media,and spread the threat actors messaging organically.The effort

211、 pays for itself when the disinformation narrative becomes endemic.Exploit the Recommendation Algorithms Disinformation actors can amplify their campaign content to new audiences by taking advantage of a platforms recommendation algorithms.Powered by ML,the recommendation algorithms infer a users in

212、terests and viewpoints to deliver tailored content.Every user interaction with a“heart”or an“angry”reaction can teach the algorithm about user preferences,a process known as“self-selected”personalization.127 The algorithm then pushes the content to another user who is either connected or is a“look-a

213、like,”fitting the profile of someone who might also like the contenta process referred by some researchers as“pre-selected personalization.”Both self-selected and pre-selected personalization can create a so-called filter bubble,a closed information loop which can increase polarization and drive use

214、rs towards fringe content.128 The volume and simultaneity of data posted to social media platforms requires them to decide what content to prioritize showing to the user,a ranking process that ML helps to manage.At the core of many social networks is a system based on deep learning recommendation mo

215、dels(DLRM)and the concept of collaborative filteringthe idea that the best recommendations come from people who have similar tastes.Collaborative filtering prioritizes the contentposts,pages,groups,eventsthat a user might like based on the users reactions and those of people with similar profiles.Fa

216、cebooks average dataset for collaborative Center for Security and Emerging Technology|34 filtering has 100 billion ratings,over 2 billion users,and millions of itemsa massive scale.129 Its Newsfeed algorithm ranks the relevance of content to the user based on multiple criteria,including personalized

217、 data from past activity,information about the posts author,and the popularity of a post.Other platforms also collect massive amounts of user data to tailor content.Platforms can tweak recommendations by altering the relative importance of various criteria.130 Most platforms that recommend content u

218、se similar deep learning architectures.131 These types of architectures can exacerbate filter bubbles that threat actors can exploit.As one former YouTube engineer explained,as an algorithm gets better at predicting who may find content engaging,it is less likely to recommend such content to those w

219、ho will not.132 Algorithmic recommendations,such as promotion of borderline and manipulative content,become harder to notice,because the content becomes less likely to reach users who might be offended and report it to the platforms.133 In some instances,recommendation engines can gradually increase

220、 the polarization of content they suggest,increasing controversy enough to keep users engaged,but not enough to be reported as violating terms of service.134 Researchers at DeepMind similarly concluded that feedback loops in recommendation systems can feed filter bubbles and create echo chambers,nar

221、rowing users exposure to diverse content and potentially“shaping their worldview.”135 Threat actors can also hack the recommender algorithms.On the less sophisticated end of the spectrum,“shilling attacks”against recommender systems can simply involve creating fake user profiles that friend authenti

222、c users or join authentic groups,amplify content with a high volume of likes and engagements,or swarm the posts supporting the narrative with positive ratings while attacking opposing views.136 These actions,in turn,increase the exposure of the target audience and their algorithmically profiled look

223、alikes to malicious content.These techniques may not use ML,but rather abuse the ML within the recommender systems.The second type of attack on the recommender systems targets“data voids,”or gaps in content returned by the search engine and Center for Security and Emerging Technology|35 social media

224、 search functions in response to specific search terms.137 Examples of data voids include strategic new terms or outdated terms.Once threat actors identify obscure search queries that return sparse content,they can create content that populates when these search terms are entered.ML techniques can h

225、elp threat actors find these voids by varying search terms and filling them using a natural language generation system to create malicious content.Other amplification techniques can drive users to search for specific terms,lending credibility to deceptive messages.Search-adjacent recommendation syst

226、ems such as auto-play,auto-fill,and trending topics are similarly vulnerable to manipulation.138 Finally,threat actors can harness adversarial approaches to amplify their content.Like other ML systems,recommendation algorithms are vulnerable to evasion attacks.139 In this type of attack,an operator

227、makes subtle changes to an input that,while imperceptible to a human eye,causes an ML system to make a mistake.In a proof of concept,security researchers have applied this technique to recommendation systems that decide what content a user sees on their social media feed.Researchers manipulated the

228、recommendation algorithms by gradually modifying genuine images to fool a classifier and cause it to make an incorrect prediction.140 In this attack,introducing small imperceptible changes to images in a post can significantly increase a posts relevance score and improve the rankings of related item

229、s by the social media recommendation algorithms.The higher ranking pushes the post with the manipulated image to more users feeds,amplifying its reach.141 In theory,similar attacks could exploit both the content and the group recommendation algorithm to make it easier for users to discover groups co

230、ntrolled by the disinformation operators.A groups cover image can be manipulated with the attack described above.For example,when Facebook adjusted its recommendation algorithms to weigh community content more heavily,it prioritized engagement with most relevant content and social interactions with

231、friends and family through groups.142 In addition to other unintended consequences,such as prioritizing polarizing content,this change also increased group discoverability,making it easier Center for Security and Emerging Technology|36 for users to find fringe groups.143 However,in response to the r

232、apid growth of conspiracy theory communities such as QAnon,the platform recently removed civic,health-related,and overtly hyper-partisan groups from its recommendation algorithm.Yet this change did not impact groups in other categories.144 Using the attack described above,threat actors can manipulat

233、e groups cover images to increase the ranking of their group by the recommendation algorithm and drive traffic to the seemingly innocuous groups they cultivate,such as private interest-based groups about tourism or food.For example,once they have built their audience on beautiful pictures of Riga,as

234、 we describe in our RICHDATA Framework report,they can repurpose the group for a future disinformation campaign.145 This technique can potentially turn innocuous private groups into self-reinforcing echo-chambers for radicalization,hard for researchers to observe and for platforms to police.146 Barr

235、ing further interventions,recommendation algorithms trained on the preferences of like-minded individuals and tuned to reinforce users leanings are likely to perpetuate polarization that threat actors can deepen and exploit.Despite recent experiments to reduce exposure to extreme and malicious conte

236、nt,147 platforms continue to state that users remain in charge of their experience.148 This means that user biasesand choices by their algorithmically profiled look-alikeswill continue to drive filter bubbles and echo chambers.Boost Conspiracy Information Laundering In 2018,Mark Zuckerberg noted tha

237、t Facebook research“suggests that no matter where we draw the lines for what is allowed,as a piece of content gets close to that line,people will engage with it more on average.”149 Users find borderline content more engaging,and recommendation algorithms are built to serve users the content they fi

238、nd most engaging,underscoring and exacerbating the participatory nature of disinformation.Center for Security and Emerging Technology|37 ML can help threat actors exploit this feature to wrap their message into engaging conspiracy theory content.Researchers studying the growth of anti-vaccination an

239、d the QAnon conspiracy communities warned that threat actors affiliated with Russia cultivated,amplified,and aligned narratives within these communities.150 In theory,trained ML models could find patterns across conspiracy narratives,to replicate and scale the type of cross-pollination that occurred

240、 between QAnon and the anti-vaccination community during the COVID-19 pandemic.151 Finally,threat actors can use natural language generation systems to produce new conspiracies and seed new narratives.Researchers have demonstrated the ability to generate synthetic text mimicking the style and conten

241、t of QAnon messages.152 Threat actors could fine-tune an open-source large language model on a dataset that contains different conspiracy theories and recent news stories,to then generate a new set of conspiracy messages that appeal to broader audiences or even replace existing players.153 Automate

242、Trolling Content generation capabilities within conversational AI technologies may soon increase the scale of trolling.Conversational AI is an umbrella term for technologies that enable machines to communicate with individual users in conversational language through voice,text,and video.They use lar

243、ge volumes of data,ML,and NLP to imitate human interactions,recognizing speech and text input and generating a response.154 Open domain chatbots may soon reduce the need for human trolls,while natural language generation can improve the output of automated trolling.These systems are conversational A

244、I agents that can engage in continuous dialog on a variety of subjects in multiple languages and accomplish decision-making tasks.155 In 2020,the Brain Team from Google Research presented Meena,a multi-turn open domain chatbot trained on data from public domain social media conversations.156 Meena c

245、ould engage in convincing conversationsshe made jokes,spoke colloquial Center for Security and Emerging Technology|38 English,and returned relevant answers to questions across a variety of topics.157 Compared to her predecessors,Meena provides a glimpse into the future of chatbots and their growing

246、sophistication.158 Conversational AI platforms are also becoming more accessible as the advanced building blocks for creating chatbots enter the open-source market.159 Many high-performing chatbots developed by well-resourced organizations for legitimate purposes are open source,such as Facebooks Bl

247、enderBot,the first chatbot model to exhibit humanlike traits,assume a personality,maintain a lengthy conversation,and show empathy.160 Researchers released the complete model,code,and evaluation set-up to drive research forward.These capabilities are also diffusing as countries race to claim suprema

248、cy in AI.Chinese technology giant Baidu recently built its own version of a chatbot.PLATO-XL,a 10 billion parameter conversational model,can engage in multi-turn dialog in Chinese and English.This diffusion is likely to continue as technology matures and costs reduce.161 For threat actors these capa

249、bilities may be hard to resist,particularly as part of a human-machine team.A chatbot drawing on advanced large language models could be a believable character on a variety of platforms with reduced need for a human operator.Using fine-tuning,threat actors could train a chatbot to specialize in spec

250、ific trolling techniques,such as provocation,nuisance,or social engineering.162 With ML tools able to“understand”online conversations and respond in natural ways,chatbots could present a variety of responses from which a human operator could select,curate output,and post a response.This would allow

251、a human operator,especially one operating in a foreign language,to manage conversation streams more efficiently and effectively.Theoretically,a human operator could run multiple“plays in three acts”simultaneously,with multiple chatbots role-playing an argument among themselves to manufacture consens

252、us or stir up controversy.163 Whatever their objectiveto support a viewpoint,to attack users expressing alternative views,or to generate engagementchatbot trolls could become increasingly Center for Security and Emerging Technology|39 difficult to distinguish from humans.164 Human-machine teaming ca

253、n reduce the labor-intensive process for the solitary human operator,help more closely impersonate authentic users,and engage more humans in online conversations through an exponential increase in scale.Rapid advancements in conversational AI could plausibly lead to fully automated trolling,removing

254、 the human operator from the loop.An automated trolling system could leverage several technologies:chatbots engaging in multi-turn conversations in comments on posts,natural language generation systems to write cogent paragraphs advancing an argument,and ML-powered sentiment analysis to assess the e

255、ffect of their messaging and optimize deployment to comment sections of specific posts.Built into the trolling chatbot,controversy detectionanother subfield of sentiment analysiscould pick up on underlying controversy not explicitly mentioned in text and help identify new wedge issues to exploit.165

256、 That said,platforms have a role in limiting some of the worst effects of these scenarios.Autonomous deployment of a trolling chatbot system on social media would require access to the platform APIs.In addition,a trolling system would have to masquerade as a legitimate application,such as an airline

257、 agent chatbot,as platforms are unlikely to allow an illegitimate trolling chatbot access to the platform.Threat actors success in this effort substantially depends on what chatbot behavior the platforms deem legitimate,and the ability of operatives to evade the platforms detection methods.Mitigatio

258、ns may be more difficult if conversational AI systems are built into the social media platforms themselves and then exploited.Scale Up Mobilization with Personalized Disinformation NLP,open domain chatbots,and deepfake technology may advance the process of mobilizing a targeted population.ML applied

259、 to the actualization stage allows the creation of more precisely targeted and personalized disinformation to cultivate Center for Security and Emerging Technology|40 influencers and inspire them to action.Threat actors can combine the various techniques we have discussed previously to enhance the a

260、ctualization of their campaign.The built-in micro-targeting features of platforms,social network analysis,and sentiment analysis of an individuals posts,could feed precisely targeted disinformation and focus it on those users that are most likely to take action.ML-enabled social network analysis can

261、 identify similarities between individuals and groups and gauge the strength of their connections.Advanced sentiment analysis and psychographics systems can help build a detailed personality profile from language.166 A stance detection model could identify users exhibiting signs of disillusionment,c

262、ynicism,susceptibility to influence,and aggression,helping operators target them for cultivation.167 One study found that violent language had an impact on“aggressive citizens,”where even mild violent metaphors significantly multiplied support for political violence in individuals displaying this tr

263、ait.168 Natural language generation systems can help human operators hone the unique lexicon of fringe communities and equip the operator with a stream of convincing radicalizing content to drive individuals to real-world action.169 Taking this a step further,advances in chatbots and deepfake techno

264、logy may soon combine to give threat actors new techniques to engage individuals convincingly and persuade them to action.Efforts like the Russian IRAs operations to organize rallies in the United States could be supercharged through personalized chatbots and deepfake impersonations as operators upg

265、rade direct messaging to engage unwitting targets over video chat.A personalized chatbot that“knows you”may soon be a tool in the disinformation operators arsenal.Chatbots are increasingly more expressive and can respond naturally using everyday language.170 By building what feels to humans like gen

266、uine relationships,chatbots are becoming companions and trusted confidantes.171 In theory,threat actors could fine-tune or draw on open-source models to build personalized chatbots that serve their cultivation purposes.Creating and training simpler chatbots is already publicly available and requires

267、 no knowledge of ML.172 Chatbots fine-tuned Center for Security and Emerging Technology|41 on the dataset of personal messages and posts of a loved one are already interacting with humans via text and audio.173 Operators could apply the same social engineering techniques used against super-spreaders

268、 to build custom datasets for lesser known targets of interest,such as grassroots activists,someone engulfed in a conspiracy theory community,or leaders of an extremist organization.Scraped public posts from targeted individuals and their close network could fine-tune a chatbot on the language patte

269、rn of an acquaintance.Set in a context of presumed trust,a personalized chatbot that sounds like a distant colleague can connect with the target through a social media direct messaging to organize events on the ground,while obscuring any red flags of the foreign disinformation operators origin.Final

270、ly,threat actors may soon have powerful options to scale up cultivation and mobilization by bringing chatbots to life with synthetic video skins for video calling.In 2020,a researcher at the BlackHat cybersecurity conference,using an open source chatbot and deepfake technologies,created a synthetic

271、clone of himself.He fine-tuned a chatbot on his own text messages,combined it with a synthetic video skin of himself,and had his“clone”carry on a video-call with a friend.174 While a glitchy prototype at the time,a similar technique could help threat actors increase the scale of their operations to

272、engage many susceptible users simultaneously and move them towards action.Technology is evolving to make impersonation possible in real time over a video call,enabling threat actors to mask their origin as they cultivate their targets with direct engagement.175 Avatarify is just one example of an op

273、en-source application that provides a digital avatar of someone elses face over ones own in video conferencing applications.176 This solution is another implementation of a head-swapping deepfake technique discussed in the earlier sections and only a few images of the target are necessary to imperso

274、nate thema sample easy to acquire from their public social media profiles.The technique automates image animation by combining the appearance extracted from the targets image to impersonate them with motion patterns derived from the real-time video of the impersonator.177 At present,Avatarify requir

275、es additional code to run and hardware with enough computing power for video-gaming.These are not Center for Security and Emerging Technology|42 obstacles to a skilled and well-resourced operation.In addition,technology is likely to evolve to run on more accessible hardware.Network lag and low light

276、ing can help mask any observable lip-syncing issues.The source-code for this app is readily available on GitHub for threat actors to experiment with and improve.178 What sounds like science fictiona chatbot that talks like a trusted colleague,an imposter donning a synthetically-generated face of a f

277、riend in a real-time video call,or some combination of bothmay soon emerge as the next set of disinformation tools.Threat actors leveraging advances in ML that enable chatbots with synthetically generated skins could become more successful in building online relationships and persuading humans into

278、action or inaction offline.Chatbots today may not yet be nuanced enough to effectively radicalize a human into creating real world effects,but this may not stay true for long.Center for Security and Emerging Technology|43 Key Findings The application of ML to disinformation operations is rapidly evo

279、lving,powered by data abundance,innovative algorithms,and massive computing power.Authoritarian regimes,notably China and Russia,are actively pursuing AI/ML research in a number of areas that will enable greater manipulation of their domestic information environments,while directing their disinforma

280、tion capabilities at adversaries outside their borders.180 Open societies are vulnerable to the malicious effects of AI-powered influence as they struggle to balance free speech with the harmful effects of disinformation designed to provoke,disrupt,and divide.Resolving this tension in the pluralisti

281、c marketplace of ideas may be increasingly difficult in the atmosphere of a fractured information environment and personalized information loops.ML technologies are likely to exacerbate the current disinformation crisis.Disinformation campaigns exploit the features of todays information environment

282、and the deeply embedded characteristics of human cognition.Presently,social media users produce and consume an immense amount of content.In 2020,humanity posted an average of 500 million tweets,viewed 4 billion Facebook videos,and created 4 billion snapsevery day.181 Human cognition is ill-equipped

283、to handle the volume of content that the average user is exposed to daily.The potential deluge of AI-powered disinformation may strain individual decision-making,societal cohesion,and the ability to reach consensus that are“If everybody always lies to you,the consequence is not that you believe the

284、lies,but rather that nobody believes anything any longer.And a people that no longer can believe anything cannot make up its mind.It is deprived not only of its capacity to act but also of its capacity to think and to judge.And with such a people you can then do what you please.”179 Hannah Arendt Ce

285、nter for Security and Emerging Technology|44 critical for democracies to function.Policy and technical countermeasures are necessary to mitigate the disruptive effects,in a whole-of-society effort.Protecting the shared public square requires an understanding of the threat of AI-powered campaigns.The

286、 below findings outline how AI/ML capabilities may exacerbate existing trends and aid disinformation campaigns,followed by recommendations to mitigate their impact.AI will augment stages of disinformation campaigns,but the gains are uneven ML can assist human operators in creating more engaging cont

287、ent and designing more precisely targeted disinformation campaigns.In the interactive stages of troll patrol and actualization,ML can currently augment the work of human operators,but full automation at scale would require significant software engineering effort,further ML advances,and nearly unfett

288、ered access to platform APIs and user data.The degree of success depends on how well operators mask their true intentions.Furthermore,ML can also assist in the reconnaissance stage by providing a more nuanced understanding of the information environment through social listening and sentiment analysi

289、s.It can help identify societal fissures and fine-tune personalized disinformation from the data that individuals and target societies post online.AI-generated fake personas will assist in the task of building infrastructure and make the detection of fake accounts and campaigns more difficult.Newer

290、ML technologies,particularly natural language generation and visual synthetic media,are now capable enough to help human operators create engaging and targeted content that can go viral.Social bots are likely to pass as humans online,while conversational AI chatbots will enhance various trolling tec

291、hniques.Donning AI-generated clones of trusted persons and armed with insights from user data,advanced multi-dialog chatbots may further personalize disinformation,engaging humans one-on-one and mobilizing them to action.From artificially generated avatars,Center for Security and Emerging Technology

292、|45 resumes,and employers to fake pets,food,and apartments,nonexistent humans are poised to live believable lives online.However,full automation at scale will require software engineering effort and further AI advances before it becomes a significant threat.Given the benefits,well-resourced actors a

293、re likely to invest in AI-enabled automation to augment human operators and increase the speed and scale of their operations.Foreign,domestic,and transnational actors may drive ML-enabled disinformation “Disinformation starts at home”is a popular axiom among disinformation researchers.Nation-states

294、often test new disinformation tactics on their domestic audiences or countries within their perceived sphere of influence before applying them more broadly.As more nations apply ML-enabled capabilities,they may test them domestically firsta trend to watch.When disinformation campaigns move abroad,th

295、ey often exploit grievances and fault lines within the targeted country.By amplifying existing content,such as conspiracy theories,they can mask their activity and further undermine societal trust.182 The lines are also blurring between foreign and domestic disinformation,as operators employ“influen

296、ce as a service”firms to mask their activity,making attributions and countermeasures more difficult.While foreign disinformation campaigns continue to command attention,domestic actors are adopting similar tactics in political campaigns.More than 65 firms in 48 countries deployed computational propa

297、ganda services on behalf of political actors spending almost$60 million since 2009.183 Of the 150 operations in 50 countries that Facebook disrupted in the past 4 years,nearly half targeted domestic audiences.184 The 2020 U.S.presidential election saw domestic political actors using inauthentic acco

298、unts and the volume of domestic disinformation dwarfing foreign influence efforts.185 AI-driven advances have attractive applications for political campaigns.Despite the lack of measurements to prove its efficacy,the case of Cambridge Analytica demonstrated a proof of concept in how ML-powered micro

299、-targeting and psychographic profiling Center for Security and Emerging Technology|46 could draw on private and public data for influence.Today,social media companies often remove election-related activity on the basis of actor behavior,such as inauthenticity,coordination and repetition of posts,rat

300、her than the content itself.186 De-platforming on the basis of what users post is reserved for specific platform policy violations,such as content inciting violence.187 While platforms are getting better at identifying inauthentic accounts,domestic actorsor the“influence as a service”transnational f

301、irms they hirecould use ML-powered language generation systems to spread disinformation through authentic users and influencers,straining content moderation efforts.188 These systems enable operators to generate a greater variety of messages to advance a talking point,avoiding copying and pasting te

302、chniques,known as“copy pasta,”that can be easily flagged by platforms as a sign of coordinated activity.In 2019,Twitters then-CEO sounded the alarm about the impact of AI on political discourse,warning:“Internet political ads present entirely new challenges to civic discourse:machine learning-based

303、optimization of messaging and micro-targeting,unchecked misleading information,and deep fakes.All at increasing velocity,sophistication,and overwhelming scale.”189 As AI-enabled content generation tools become more widespread,the temptation to use them by candidates or elected officials will only gr

304、ow,leaving social media platforms in an unenviable position of having to make nuanced moderation decisions about a high volume of protected political speech and risking accusations of bias.While political speech is protected in the United States,questions remain about whether these protections shoul

305、d extend to AI-generated or bot-amplified speech.With a broad interpretation of protected speech,AI-powered disinformation at scale has the potential to reach large audiences and become endemic.Developing norms to discourage the use of artificial amplification by political campaigns and their affili

306、ates during elections is not a panacea,but a necessary start.Center for Security and Emerging Technology|47 Social media platforms and ML applications serve as both the battlegrounds and the weapons of disinformation campaigns Social media platform features define the terrain and shape threat actors

307、 tactics.Disinformation operators will seek to misuse social media platforms and their features,exploiting their design that helps people to connect and engage.As long as market incentives drive growth over security,there is a risk that the architecture of established and new platforms will be initi

308、ally insecure and difficult to harden later.Adding to the challenge,many ML applications are dual-use.One persons humorous synthetic video is anothers deepfake.These technologies are driving innovation in a variety of industries,from increasing the supply of stock photos for digital marketing to bri

309、nging life-like avatars into virtual environments for war-fighter simulations and remote workspaces.However,deepfakes are now used in head-swapping apps for nonconsensual pornography,as a tool of political warfare,and to subvert detection of fake accounts by platforms.NLP in chatbots augments servic

310、e centers and increases the efficiency of virtual assistants.However,it also opens the door to increasingly advanced chatbots that carry out precision trolling.ML systems using advanced psychographic targeting help advertisers expand their audiences and enable political campaigns to identify undecid

311、ed voters.Malicious actors can use these same tools to micro-target and mobilize unwitting audiences in a disinformation campaign.The open nature of AI research provides avenues for misuse The open-source culture of the AI research community has fostered many recent advances,but it also provides rea

312、dy access for malicious actors.Top AI research institutions and independent researchers post source code to GitHub to share with others.190 There are tutorials about how to fine-tune the open-source code to generate synthetic text with a specific tone,or how to create a lip-synching video of yoursel

313、f speaking 30 different languages.191 The many benefits of open-sourcing include generating public awareness,enabling researchers to build on shared research to drive innovation,and generating training data to improve deepfake Center for Security and Emerging Technology|48 detection systems.However,

314、efforts to increase public interest in technology,such as a website with images of convincing GAN-generated fake people,provide less sophisticated threat actors ready access to this technology.192 Threat actors can fine-tune open-source ML models to produce hyperrealistic images or clone voices of s

315、pecific individuals with precisely curated small training datasets with relatively low computing requirements.193 This tension is reminiscent of the offense-defense debate in the cybersecurity research community.The default assumption of those who advocate for openly publishing nearly ready-to-use M

316、L models is akin to security researchers releasing information about newly discovered vulnerabilities and publishing exploit code to force software vendors to fix them.194 Both are motivated by the goal of improving security and advancing research,yet they can invite threat actors to adopt the tools

317、 and exploit the vulnerabilities,leaving unwitting users vulnerable until defenses catch up.Disinformation campaigns cross platforms,while technical and policy countermeasures are platform-specific Disinformation campaigns in their present form are asymmetric.Threat actors often operate across many

318、platforms making it more difficult to understand the full scope of activity.Companies work independently to counter malign activity on their platforms;however,building visibility of the larger campaign requires greater coordination among targeted entities.Despite some automated threat detection on l

319、arger platforms,threat hunting remains a manual and labor-intensive process that relies on tips from the government,the media,and civil society partners.Silos between platforms,a lack of standardization on sharable information,and the varying degrees of threat monitoring and self-policing between la

320、rge and small platforms impede discovery,neutralization,and attribution of cross-platform campaigns in real time.In addition,the lines are blurring between the foreign-manufactured disinformation and the homegrown misinformation ecosystems.Threat actors increasingly seed their payloads on small frin

321、ge platforms or entirely“off platform”on their own websites,where it is below the radar of the larger platforms threat Center for Security and Emerging Technology|49 hunting teams.From there,the disinformation finds its way into the mainstream platforms,grows,and retreats to the fringe to regroup af

322、ter takedowns by major platforms.While large platforms deploy AI-enabled tools and human threat investigators to detect behavior that violates their policies,smaller platforms often lack the requisite tools,human capacity,or the will to police content for harmful use.AI-enabled recommendation algori

323、thms can fuel personalized disinformation By tailoring each users information diet based on their past engagements and the actions of algorithmically determined similar profiles,recommender algorithms can move their users into closed-loop information systems further exacerbating polarization.195 Ove

324、r time recommendations lose their effectiveness and require continuous intervention to prevent the narrowing of the pool of content presented to the user.Recommendation algorithms can also be manipulated by disinformation actors to exacerbate information silos,driving users deeper into extreme group

325、s and conspiracy theories.While social media platforms battle this trend with various technical solutions,it remains difficult to independently verify if their methods are working.196 The scale of todays information environment means that platforms must make choices about what data to surface to the

326、ir users.At the crux of the problem is the tension between putting users in charge of their experience and adjusting the algorithmic architecture that exacerbates those choices.Traditional media are targets of AI-powered disinformation campaigns Media coverage of disinformation,misinformation,and ha

327、cked-and-leaked information plays a critical role in its virality and reach.A notable example is the media amplification of hacked and leaked Democratic National Committee and Clinton campaign emails.197 Because of the reach,media organizations are prime targets of disinformation campaigns.However,t

328、hey often lack the necessary tools to check the authenticity of ML-generated information and its Center for Security and Emerging Technology|50 sources.Synthetic media can further obscure authorship,making the task more daunting.While government organizations and technology companies are developing

329、tools to detect synthetic image,video,and audio,few capabilities exist for the detection of machine generated text.Some media organizations have begun to put in place safeguards to avoid becoming super-spreaders of disinformation,but this practice is not universal.Digital media literacy is critical

330、in countering ML-powered disinformation While companies attempt to detect disinformation operations on their platforms,a substantial amount of malicious content still reaches its intended audience.Most solutions focus on mitigating the supply of disinformation.Relatively less effort has focused on t

331、he demand-sidethe cognitive biases,individual grievances,and digital media literacy of individual users.The factors that increase societal vulnerability to disinformation require urgent attention.Chief among them is the need to build societal resilience and awareness about ML-generated content and t

332、he techniques disinformation operators use to exploit users data in order to shape their perception of reality.Center for Security and Emerging Technology|51 Recommendations Jake Sullivan,President Bidens national security advisor,recently warned of the new challenges of disinformation and cyber int

333、rusions from the technologies once thought would almost inevitably favor democratic values.Sullivan said,“if democracies dont turn back this tide,the second phase of the digital revolution will grow darker with the proliferation of autonomous disinformation.”198 The challenges engendered by these campaigns were already present without the addition of ML techniques which is why the United States m