《CSET：2024年混合AI事件報告框架論證：其他行業事件報告系統中的經驗教訓總結報告（英文版）（49頁）.pdf》由會員分享，可在線閱讀，更多相關《CSET：2024年混合AI事件報告框架論證：其他行業事件報告系統中的經驗教訓總結報告（英文版）（49頁）.pdf（49頁珍藏版）》請在三個皮匠報告上搜索。

1、Issue BriefMarch 2024An Argument for Hybrid AI Incident ReportingLessons Learned from Other Incident Reporting SystemsAuthorsRen Bin Lee DixonHeather Frase Center for Security and Emerging Technology|1 Executive Summary AI incidents have been occurring with growing frequency since AI capabilities be

2、gan advancing rapidly in the last decade.Despite the number of incidents that have emerged during the development and deployment of AI,there is not yet a concerted U.S.policy effort to monitor,document,and compile AI incidents and use the data to enhance our understanding of AI harm and inform AI sa

3、fety policies in order to foster a robust AI safety ecosystem.In response to this critical gap,the objectives of this paper are to:Examine and assess existing AI incident reporting initiativesboth databases and government initiatives.Elicit lessons from incident reporting databases from other sector

4、s.Provide recommendations based on our analysis.Propose a federated*and standardized hybrid reporting framework that consists of Mandatory reporting:Organizations must report certain incidents as directed by regulations,usually to a government agency.Voluntary reporting:Individuals and groups are pe

5、rmitted and encouraged to report incidents,often with clear guidelines and policies,and usually to a government agency or professional group.Citizen reporting:This is similar to voluntary reporting,but incidents are reported by the public,journalists,and organizations acting as watchdogs.*For the pu

6、rpose of this paper,we define a federated framework as a centralized framework prescribed by a singular authoritative government body or the federal government.The framework stipulates a set of minimum requirements that can be adapted and implemented across government agencies or non-governmental or

7、ganizations.Center for Security and Emerging Technology|2 When discussing incident reporting in this paper,we emphasize reporting to an independent external organization(e.g.,a government agency,professional association,oversight body,etc.).A survey of existing AI incident collection efforts identif

8、ied only two citizen-reporting organizations actively capturing AI incidents.Additionally,a review of AI legislative initiatives globally revealed China,the European Union,Brazil,and Canada have enacted or proposed guidelines for mandatory AI incident reporting.Currently,there arent any significant

9、legislative initiatives for establishing an AI incident reporting policy framework in the United States.The available U.S.governmental documents that mention reporting AI incidents are recommendations and guidelines for implementing reporting mechanisms but not necessarily toward an external entity.

10、Looking at incident reporting frameworks from the healthcare,transportation,and cybersecurity sectors yielded valuable lessons.The healthcare sectors use of voluntary reporting resulted in missing incidents and incomparable data points for analysis.The transportation sector has an established incide

11、nt reporting framework that includes investigative boards for identifying root causes,which are then used to inform evidence-based safety measures.In cybersecurity,the U.S.government has issued a series of mandates requiring mandatory reporting in selected domains,shifting away from relying on stand

12、ards and other soft laws.Our analysis of the two AI incident reporting databases,emerging government initiatives related to AI incident reporting,and the various incident reporting systems in the healthcare,transportation,and cybersecurity sectors revealed disadvantages and advantages.These insights

13、 offered several important lessons that can be applied to an AI incident reporting policy framework,as discussed in the following:Limited incident reporting frameworks are inadequate.Across the board,the incident reporting initiatives examined in this paper often emphasized either citizen,voluntary,

14、or mandatory reporting,typically focusing on one or two of these reporting categories.In isolation,each of these three frameworks has limitations.Inconsistent data creates meaningless data.Relying on state initiatives or domain-specific guidelines will likely produce uneven or inconsistent data that

15、 Center for Security and Emerging Technology|3 might not be adequate for aggregating AI incident data for statistical analysis or accurately depicting the many dimensions of AI harm.There is a need for a federated AI incident reporting framework.The absence of a federated AI incident reporting polic

16、y framework has impacted incident data collection efforts in the healthcare sector,resulting in fragmented and inconsistent reporting initiatives.Incident investigation supports effective safety policies.An investigative safety board can be useful for conducting root-cause analysis of significant AI

17、 incidents and providing feedback to help AI actors improve their design and development,enable policymakers to craft effective regulations,and educate the public on AI safety.*Based on the observations discussed above and the nature of AI as a general-purpose technology,we make the following recomm

18、endations to address the current gap in AI incident reporting.Establish clear policies for a federated hybrid reporting framework.Policymakers should establish a federated and comprehensive AI incident reporting policy framework to gather incident data across sectors and applications.AI incidents sh

19、ould be reported to an independent external entity(e.g.,government agency,professional association,oversight body,etc.)to promote transparency and accountability in AI incident management.A hybrid reporting framework is supported by:Mandatory reporting:Relevant AI actors should be mandated to report

20、 covered incidents in a timely manner.*UNESCO defines AI actors as any actor involved in at least one stage of the AI system lifecycle,and can refer both to natural and legal persons,such as researchers,programmers,engineers,data scientists,end-users,business enterprises,universities and public and

21、private entities,among others.See:“Recommendation on the Ethics of Artificial Intelligence,”UNESCO(2021),10,https:/unesdoc.unesco.org/ark:/48223/pf0000381137.Center for Security and Emerging Technology|4 Voluntary reporting:Voluntary reporting frameworks should also be established alongside the mand

22、atory framework to capture AI incidents outside the mandatory jurisdiction.Citizen reporting:An easily accessible reporting framework should be made available for the public and all other stakeholders to report and document AI incidents.Develop a standardized and authoritative classification system.

23、The AI incident reporting framework should include a standardized set of disclosed information plus accommodations for the unique characteristics of distinct domains,such as privacy concerns and other regulatory requirements.Create an independent AI incident investigation agency.When a significant A

24、I incident occurs,an independent board should investigate the root cause and provide evidence-based safety recommendations.Explore automated data collection mechanisms.Automated data collection mechanisms could be highly advantageous to obtain technical and contextual information from AI incidents.F

25、urther research will be needed to explore the necessary content and considerations for implementing a comprehensive reporting framework that is applicable across sectors and applications.We will explore this in a follow-up paper and will not delve into it in this paper.The ability to mitigate AI har

26、ms and manage their aftermath competently can shape public conversations about AI usage.An AI incident reporting framework must be integrated as an essential component of AI safety rather than developed as an afterthought in AI legislative initiatives.The present moment offers a prime opportunity to

27、 establish an AI incident reporting framework with relatively low stakes.However,this window is rapidly closing as AI becomes more prevalent across applications and sectors.A federated,comprehensive,and standardized framework will prevent data gaps and enhance data quality.Adopting a hybrid framewor

28、k that includes mandatory,voluntary,and citizen reporting will improve data fidelity,providing a more accurate representation of the emerging trends in AI harm and risk.Center for Security and Emerging Technology|5 Table of Contents Executive Summary.1 Introduction.6 A Brief Overview of Incident Rep

29、orting.10 Current AI Incident Reporting.11 Government Initiatives for AI Incident Reporting.15 Lessons from Incident Reporting Policies in Healthcare,Transportation,and Cybersecurity Sectors.20 Discussion.31 Recommendations.35 Conclusion.39 Authors.41 Acknowledgements.41 Endnotes.42 Center for Secur

30、ity and Emerging Technology|6 Introduction The potential capabilities and benefits that AI can bring are immense and wide-reaching.Notably,the technology has contributed to significant advancements in foundational scientific research that otherwise would have taken years or decades to achieve.For in

31、stance,in bioscience and medicine,researchers used AI to accurately predict protein structures that could accelerate new medical discovery and efficiently identify potential compounds that could attack an antibiotic-resistant superbug.1 In clean energy,a reinforcement learning algorithm has successf

32、ully controlled nuclear fusion plasma in a tokamak(a machine that controls and contains heated hydrogen plasma),accelerating toward clean energy generated from nuclear fusion.2 That said,as AI applications and systems have become more prevalent across sectors and industries,the number of AI incident

33、s has increased.An AI incident can be generally described as an event where an entity experienced tangible or intangible harm that can be directly linked to a consequence of the behavior of an AI system.3 Several notable recent incidentssuch as biased outcomes in AI systems used to predict recidivis

34、m,in facial recognition technology,hiring decisions,and welfare allocation decisionshave drawn significant public attention to the issue of AI harm.4 The data captured in the AI Incident Database shows a rapid growth in AI incidents since 2010(see Figure 1).*AI incidents are submitted by community m

35、embers to the AI Incident Database before expert review and inclusion into the database.Data is gathered from the October 2023 database backup,the most recent available at time of writing.Center for Security and Emerging Technology|7 Figure 1.Cumulative AI Incidents by Year Center for Security and E

36、merging Technology|8 A Snapshot of AI Incidents from the AI Incident Database Between May and June 2023,volunteers tracking AI incidents submitted thirteen AI incidents to the AIID.The following examples are a snapshot of some of the incidents,to highlight the types of impacts they can have:The Nati

37、onal Eating Disorders Association in the United States took down its artificial intelligence chatbot“Tessa,”which developers designed to provide healthy eating tips.Contrary to its developers expectations,the chatbot reportedly offered bad eating advice that could harm people seeking help.5 A fake A

38、I-generated image showing a building near the Pentagon exploding circulated widely on social media.A financial news site reported the image,which caused a brief dip in the U.S.stock market before experts dispelled the image as fake.6 The U.S.National Highway Traffic Safety Administration has been in

39、vestigating several road accidents and fatalities linked to Tesla Autopilot.These incidents have increased significantly from 2019 to 2023,reaching more than 736 reported crashes.7 Presently,dedicated AI incident reporting databases are primarily citizen reporting.It is ad hoc,in the beginning stage

40、s,dependent upon volunteers,and funded by private donations.*Despite the number of reported incidents that have emerged from the development and deployment of AI,there is not yet a concerted policy effort to document and compile AI incidents and use these data to enhance our understanding of AI harm

41、,inform AI safety policies,and in general foster a robust AI safety ecosystem.*The purpose of this report was to identify the gaps in federated AI incident reporting,thus we only focused on databases that collect AI incidents indiscriminately.We acknowledge that there may be AI incidents that are ca

42、ptured in sector-relevant databases,such as medical instruments,vehicle accidents,and hiring systems.Center for Security and Emerging Technology|9 In response to this critical gap,the objective of this paper is to:Examine and assess existing AI incident reporting initiativesboth databases and govern

43、ment initiatives.Elicit lessons from incident reporting databases from other sectors.Provide recommendations based on our analysis.Propose a federated,comprehensive,and consistent reporting framework that consists of mandatory,voluntary,and citizen reporting.Implementing an operational AI incident r

44、eporting framework will require in-depth assessments to determine,for example,the type of incidents to be reported,a comprehensive and adaptable classification system,the types of data that should be collected,and how data will be shared and used.Additional research will be needed to uncover this in

45、formation,which will be the focus of a follow-up paper and will not be discussed here.Overall,promoting a robust safety ecosystem for safeguarding society in the face of AI advancement is imperative to enable us to harness its full potential while minimizing the risk of associated harm.Establishing

46、an AI incident reporting policy framework can enhance our understanding of AI harm,contributing to greater AI safety and risk mitigation efforts.Center for Security and Emerging Technology|10 A Brief Overview of Incident Reporting Incident reporting has been an integral component of safety practices

47、 across different sectors to document data when harm has occurredfrom healthcare to aviation,manufacturing to occupational safety,and utilities to food safety.When adverse events or harm occur,vital data is collected to help us gain deeper insights into the root causes,uncover trends,and prevent pas

48、t failures from reoccurring.These insights,in turn,serve as a foundation for developing more accurate and effective policies that foster a robust safety framework and ecosystem.In this paper,we focus on incident reporting to an independent external organization(e.g.,government agency,professional as

49、sociation,oversight body,etc.).We do not consider internal incident collections(e.g.,company software bug tracking,internal help desk tickets,etc.)or third-party reports that companies collect on their products(e.g.,customer complaints,customer support emails,etc.).Incident reporting to third-party

50、organizations can fall into three main categories:Mandatory reporting:Organizations must report certain incidents as directed by regulations,usually to a government agency.Voluntary reporting:Individuals and groups are permitted and encouraged to report incidents,often with clear guidelines and poli

51、cies,and usually to a government agency or professional groups.Citizen reporting:This is similar to voluntary reporting,but incidents are reported by the public and organizations acting as watchdogs.The choice between implementing voluntary or mandatory reporting systems varies across sectors and in

52、dustries,as do the disclosed information and criteria for reporting.Mandatory reporting requires obligated actors to report covered incidents,and non-compliance may result in legal consequences.Conversely,in voluntary reporting,organizations and relevant actors are encouraged but not required to rep

53、ort incidents.Both policy approaches have advantages and disadvantages,and various sectors have adopted different policies to address their needs.Evaluating the effectiveness of different incident reporting policy frameworks is crucial to gauging how comprehensively and reliably they can capture the

54、 full dimensions and extent of AI harm.Center for Security and Emerging Technology|11 Current AI Incident Reporting Reporting and tracking AI incidents will play a crucial role in understanding AI harms,facilitating the development of effective policy tools and measures to mitigate these harms,and r

55、educing the potential risks associated with AI.Present AI incident reporting databases are primarily based on citizen reporting.Citizen reporting is invaluable,but insufficient to exhaustively and reliably document incidents.One challenge is that the data gathered from different incident reporting f

56、rameworks have different structures,making it difficult to compare data points across the different databases and time-consuming to analyze and extract meaningful information.Furthermore,noticeable gaps exist within the dataset.Since public citizens voluntarily report incidents,there is no guarantee

57、 that all incidents are captured.Policies for establishing mandatory and voluntary incident reporting will be necessary to address these data deficiencies.A survey of existing AI incident reporting databases that track and collect information on AI incidents yielded less than a handful of key player

58、s:AI Incident Database(AIID)8 AI,Algorithmic,and Automation Incidents and Controversies Repository(AIAAIC)9 AI Vulnerability Database(AVID)10 AI Litigation Database11 The AVID emphasizes identifying vulnerabilities*in AI systems,while the AI Litigation Database focuses on documenting AI-related lega

59、l cases.12 As a result,AIID and AIAAIC are the only two key players in AI incident databases that attempt to actively capture all publicly available data related to AI harms and issues.Independently founded and operated,these two databases rely on public submissions of media reports covering AI inci

60、dents.However,they have developed different classification *The AVID defines vulnerability as any weakness in AI systems that can cause incidents.Center for Security and Emerging Technology|12 frameworks,which hamper comparable and complete documentation of all AI incidents.AI,Algorithmic,and Automa

61、tion Incidents and Controversies Repository The AI,Algorithmic,and Automation Incidents and Controversies Repository is an independent collection of incidents and controversies about AI and AI-related technologies that started in June 2019.13 As of July 2023,the Repository had more than 1,100 entrie

62、s on incidents and controversies relating to AI,algorithms,and automation.14 The AIAAIC Repository is maintained by an editorial team consisting of contributors who identify incidents and process public submissions of media reports using a six-step framework:detect,assess,classify,summarize,approve,

63、and publish.Reports are assessed based on relevance,impact,credibility,and volume before being added to the Repository.The AIAAIC Repository displays incidents on a Google sheet where users with various access levels can view data,modify data,and provide comments,making the Repository a live databas

64、e.The repository excludes reports involving certain technologies and issues,such as geopolitical issues,legislations and standards,and quantum computing.Notably,artificial general intelligence and artificial superintelligenceboth AI-related topicsare also on the exclusion list,presumably because the

65、y do not currently exist or are considered hypothetical concepts.The AIAAIC analysis of harm reflects an organizational viewpoint in which the negative impacts caused by AI systems occur either internally or externally of the organization that developed or deployed the AI system.External harms are n

66、egative impacts on individual users or stakeholders,society,and the environment,whereas internal harms affect the business reputation,operations,finances,and compliance of the organization that developed or deployed the AI system.AI Incident Database The AI Incident Database(AIID)started in May 2018

67、 and was launched publicly in November 2020.15 The database is sponsored by the UL Research Institutes,an independent safety science organization with a global reach.16 The Responsible AI Center for Security and Emerging Technology|13 Collaborative,an organization chartered to oversee the incident d

68、atabase,edits the content in the AIID.The AIID collects and catalogs AI incidents through the public submission of media reports covering a wide variety of AI incidents.As of September 2023,there were 2,813 incident reports connected to 547 unique incidents in the database.Each incident may have one

69、 or multiple reports published on the database,reported by various media outlets and providing diverse viewpoints on incidents.Submissions are reviewed and indexed by internal and volunteer editors before being published on the online database.Incidents are currently annotated with two taxonomies:th

70、e Goals,Methods,and Failures taxonomy and the Center for Security and Emerging Technologys AI Harm Taxonomy.17 Key Takeaways:Current AI Incident Databases While AIIDs and AIAAICs work is valuable in setting the early foundation and infrastructure for documenting AI incidents,both initiatives have de

71、veloped separate taxonomies and classifications for defining AI harms.For the most part,the databases emphasize and collect different information for each incident.Their conflicting definitions of harm and risks obstruct drawing parallels between the databases that could have contributed to comparab

72、le research in AI safety.See Figure 2 for an overview of the key takeaways from current AI incident databases.Center for Security and Emerging Technology|14 Figure 2.Key Takeaways:Current AI Incident Databases Center for Security and Emerging Technology|15 Government Initiatives for AI Incident Repo

73、rting After reviewing the existing AI incident reporting databases,we surveyed emerging AI government initiatives from around the world to assess their provisions for reporting AI incidents and identify potential gaps to address.Of the countries we surveyed,China was the only country that has promul

74、gated AI-related rules that include provisions for incident reporting,while the European Union,Brazil,and Canada have proposed legislative initiatives that include provisions for incident reporting.China In the last two years,China has released a series of AI-related rules to address the emerging ha

75、rms and risks associated with AI.In 2022,the Provisions on the Management of Algorithmic Recommendations in Internet Information Services and the Provisions on the Administration of Deep Synthesis Internet Information Services came into effect to regulate algorithmic recommenders and deepfakes.18 In

76、 2023,its Interim Measures for the Management of Generative Artificial Intelligence Services came into effect to mitigate the rising concerns over generative AI.19 Within these three AI-related rules,AI service providers are required to report any violations to relevant authorities,as well as establ

77、ish reporting mechanisms for the public to lodge complaints and provide feedback on their services.European Union,Brazil,and Canada Meanwhile,legislative proposals from the European Union,Brazil,and Canada include requirements for specific AI actors(AI developers,research labs,companies,organization

78、s,and operators)to report incidents to relevant authorities.The crux of the Proposal for Laying Down Harmonised Rules on Artificial Intelligence(EU AI Act)is its risk-based approach that classifies AI systems into unacceptable-risk,high-risk,limited-risk,and minimal-risk systems.20 Regarding AI inci

79、dent reporting,the EU AI Act requires developers of high-risk AI systems to report any serious incidents or malfunctioning to the corresponding authorities in the Member States where they occurred.A serious incident or malfunctioning constitutes a violation of fundamental rights in the European Unio

80、n.Reports must be made immediately or within 15 days of Center for Security and Emerging Technology|16 the incident.Conversely,providers of minimal-risk systems are encouraged to follow voluntary codes of conduct instead of mandatory obligations.High-Risk AI Systems Under the Proposed EU AI Act 1.Bi

81、ometric identification and categorization of natural persons 2.Management and operation of critical infrastructure 3.Education and vocational training 4.Employment,workers management,and access to self-employment 5.Access and enjoyment of essential private and public services and benefits 6.Law enfo

82、rcement 7.Migration,asylum,and border control management 8.Administration of justice and democratic processes Source:Proposal for Laying Down Harmonised Rules on Artificial Intelligence,Annex III Brazils draft legislation on AI regulation reflects the EU AI Act,using similar provisions.The Brazilian

83、 proposal entails reporting obligations imposed on both providers and operators to inform authorities of severe incidents that pose risks to human life,critical infrastructure,property,environmental damage,and infringements upon fundamental human rights.21 Similarly,in 2022,Canadas proposed Artifici

84、al Intelligence and Data Act outlined requirements for individuals responsible for high-impact systems to notify the Ministry of Innovation,Science,and Economic Development in situations where the system has caused substantial harm or presents a significant likelihood of causing such harm.22 Center

85、for Security and Emerging Technology|17 United States In 2023,the U.S.government announced several AI policy initiatives that included AI incident reporting.Executive Order 14110 directed the identification,collection,and investigation of AI incidents emerging from the healthcare sector and incident

86、s related to intellectual property.23 Additionally,the Executive Order instructed the Department of Homeland Security to establish an AI Safety and Security Board to provide the government with recommendations for incident response related to AI usage in critical infrastructure.Following the Executi

87、ve Order,the National AI Advisory Committee released its recommendations for piloting an adverse AI event reporting system.24 NAIAC focused its recommendations on reporting the most concrete and severe events,such as those involving national security risk,substantial injury and damage,and death to e

88、xisting regulatory authorities.Before the Executive Order was published,the U.S.National Institute of Standards and Technology(NIST)released the AI Risk Management Framework that includes a set of voluntary guidelines for sharing incident data among relevant AI actors and affected communities.25 In

89、July of the same year,the White House announced it had secured voluntary commitments from the seven leading AI companies in the United StatesAmazon,Anthropic,Google,Inflection,Meta,Microsoft,and OpenAIthat include enabling third-party discovery and reporting vulnerabilities in their AI systems.The l

90、ast two initiatives did not specify reporting incidents to an external entity.26 Key Takeaways:Government Initiatives on AI Incident Reporting Despite outlining obligations for AI developers and providers to report incidents,the legislative initiatives from China,the European Union,Brazil,and Canada

91、 did not include clear recommendations for implementing consistent federated incident reporting frameworks and data collection.A plausible implication could be that organizations that collect AI incidents might emphasize collecting different data types,and discrepancies might appear in their data ma

92、nagement.The rules from China were the only legislative initiative that addressed citizen reporting on AI incidents.The European Union,Brazil,and Canada proposals did not extend incident reporting provisions to other stakeholders,such as the public,that could potentially experience AI harm.Neverthel

93、ess,the reporting obligations outlined Center for Security and Emerging Technology|18 in the rules from China emphasize mitigating illegal activities and rule violations,which could limit the scope of incidents that get reported.For instance,system vulnerabilities and adversarial attacks might not f

94、all within this scope,and data collection on these incidents would be absent,leaving a key dimension of AI harms missing from the overall picture.The Executive Order demonstrates the U.S.governments intent to capture AI incidents.However,the Executive Order primarily covers AI incidents involving cr

95、itical infrastructure,IP,and healthcare,which excludes AI incidents that occur in other domains.NAIACs recommendations to report AI incidents to existing regulatory authorities could result in incidents falling in between regulatory gaps,given the general-purpose nature of most AI capabilities.Moreo

96、ver,NAIACs narrow focus on reporting tangible harm excludes intangible harm that is equally impactful and commonly occurring,such as bias and discriminatory decisions resulting in differential treatment.The current voluntary approach in the United States does not guarantee all relevant AI actors wil

97、l implement reporting mechanisms,or utilize one that is comprehensive and consistent across organizations.Furthermore,reporting guidelines outlined in both documents do not suggest collecting and reporting AI incidents to an independent external entity.The reporting mechanisms proposed in these guid

98、elines suggest that incident data is funneled back into the companies and organizations of the AI actors,impeding transparency and accountability when AI harm occurs and limiting information-sharing on AI vulnerabilities.It is undetermined what regulations and policies on reporting AI incidents the

99、U.S.government will announce in the coming years.For an overview of the key takeaways from government initiatives on AI incident reporting,see Figure 3.Center for Security and Emerging Technology|19 Figure 3.Key Takeaways:Government Initiatives on AI Incident Reporting Note:CN=China,EU=European Unio

100、n,BR=Brazil,CA=Canada,and US=United States.Center for Security and Emerging Technology|20 Lessons from Incident Reporting Policies in Healthcare,Transportation,and Cybersecurity Sectors To understand which type of incident reporting policy framework would work best for recording AI incidents,we look

101、 to learn lessons from incident reporting in other fields.We chose three high-risk sectors that have established incident reporting frameworks:Healthcare Transportation Cybersecurity In each sector,we analyze its reporting structure,policy evolution,and the impact of its distinct policy approach on

102、the outcomes of its incident reporting initiatives.Table 1 provides an overview of these three sectors various incident reporting systems.Center for Security and Emerging Technology|21 Table 1.Overview of Incident Reporting in Healthcare,Transportation,and Cybersecurity Center for Security and Emerg

103、ing Technology|22 Healthcare:Inconsistent Incident Reporting Background on Healthcare Incident Reporting The impetus for systematic tracking and learning from medical errors gained momentum following the publication of the pivotal book To Err is Human by the Institute of Medicine in 2000.27 The book

104、s revelation that up to 98,000 hospital deaths resulted from medical errors annually brought attention to the issue as a pressing public concern that demanded immediate policy intervention.The Centers for Disease Control and Prevention estimates about 1 in 31 hospital patients each day acquires heal

105、thcare-associated infection,costing billions of dollars in added expenses to the U.S.healthcare systemand around half of those cases could be preventable.28 Agencies and organizations established numerous incident reporting frameworks to mitigate medical errors financial and human costs.Medical erro

106、rs were tracked at various levels:organizational-based,state-based,and national-based.These frameworks range from mandatory reporting to voluntary,and are endorsed by a variety of independent and government-led organizations.Despite the severity of medical errors and their significant occurrence rat

107、e,incident reporting in healthcare is incongruous across the United States.There is considerable variation in the types of events that are mandatory to report,along with disclosure requirements.Below,we examined a selection of the more commonly used frameworks.The National Quality Forum(NQF)is a non

108、profit organization that developed Serious Reportable Events(SREs)in healthcare,which is a set of definitions and standards that some states have used to implement their own mandatory reporting systems.29 The Agency for Healthcare Research and Quality(AHRQ)is a government agency that possesses a rep

109、ository of patient safety incidents reported voluntarily by entities registered under its Patient Safety Organization Program.30 Center for Security and Emerging Technology|23 The Joint Commission is an accreditation organization that launched its sentinel event*reporting system to document adverse

110、patient incidents and encourage accredited organizations to report these events.31 The absence of a clear,federally mandated central reporting system for serious healthcare events has contributed to inconsistent efforts in documenting adverse outcomes in patient safety.32 The NQF developed a set of

111、voluntary standards(SREs)that states can adopt in their incident reporting frameworks.33 More than half the states and the District of Columbia have implemented mandatory reporting based on the SREs standards,and yet there are still discrepancies in how they utilize,implement,and view the reporting

112、of different patient safety events.The critical variations in implementing SREs within states mandatory reporting frameworks are:State-defined lists do not include any of the language within NQFs SREs,but may use NQFs standards or others as a launching pad.Modified NQF lists reference the SREs but a

113、dd,remove,or modify NQFs events or definitions.A list can be classified as“modified”even by removing one SRE.NQFs SREs are used entirely and exactly as written for creating legislation.Voluntary reporting systems can suffer from underreporting,resulting in databases that do not accurately capture th

114、e full spectrum of prevalent safety issues.As a result,the precision of incident trend analysis is also diminished.34 The AHRQ manages the Network of Patient Safety Databases,a repository of patient safety incidents reported voluntarily by entities registered under the Patient Safety Organization Pr

115、ogram.However,there are currently only 103 registered providers in the Program,which is a tiny fraction of the 6,129 hospitals in the United States.35 Due to the limited number of registered providers and the voluntary nature of their reporting system,the AHRQ admits that their database“does not con

116、tain a representative sample of patient safety concerns and cannot be used to calculate the actual incidence or prevalence of patient safety events.”36 *A sentinel event is a patient safety event that results in death,permanent harm,or severe temporary harm.Center for Security and Emerging Technolog

117、y|24 Additionally,there is a staggering difference between the number of incidents recorded in a mandatory reporting system and a voluntary system when comparing The Joint Commissions sentinel event reporting(voluntary)and the New York Patient Occurrence Reporting and Tracking System(NYPORTS)(mandat

118、ory).An analysis from 2005 found that NYPORTS recorded 11,028 adverse events between 1998 and 2003,while the national voluntary reporting system run by The Joint Commission collected a mere 176 incidents from the state within a similar timeframe.37 The significant gap in reported incidents between m

119、andatory and voluntary systems raises questions about the efficacy of a voluntary framework as a reliable mechanism for improving safety practices and its ability to represent occurring harms accurately.Voluntary reporting systems are likely to miss valuable data needed to inform and improve safety

120、measures.38 To underline the usefulness of mandatory reporting,New York Statewhich requires mandatory reportinghas used data from its NYPORTS database to formulate protocols that reduce incident occurrences.For instance,data analysis of wrong-patient/wrong-site events(the severe error of performing

121、a medical procedure on the wrong patient or performing surgery on the wrong place of the body)led to new protocols in 2001 that helped reduce such incidents from 25 events in 2002 to 17 events in 2003 in New York State.39 Transportation:Investigative Data Collection Transportation-related safety iss

122、ues in the United States are primarily overseen by the National Transportation Safety Board(NTSB)and the National Highway Traffic Safety Administration(NHTSA).All significant accidents and crashes in aviation,highways,marine,railroads,pipelines,and hazardous materials must be reported to the NTSB,wh

123、ich then carries out investigations to identify the root causes.40 The NTSB utilizes information gathered from automated data-collecting sensors and event recorders in aircraft,cars,and vessels to assist in their investigations.These automatic data collection mechanisms record crucial technical and

124、contextual information that can help identify the root causes of incidents.The NTSB utilizes the acquired data and results from its investigations and research to construct its Most Wanted List:a compilation of safety recommendations to prevent Center for Security and Emerging Technology|25 accident

125、s,reduce injuries,and save lives.41 For instance,presently,the NTSB has highlighted the need for standardized alcohol and drug testing to prevent impairment-related crashes on highways,and a ban on personal electronic devices while driving to prevent distracted driving.The NTSB also advocates for th

126、ese recommendations in state legislation,proposes regulatory amendments,suggests procedural adjustments by operators,and urges professional associations to inform their members about relevant safety issues.While the NTSB doesnt have the authority to require recipients to implement their safety recom

127、mendations,the NHTSA can enforce safety standards and regulations.42 Using its authority,the NHTSA has issued mandatory reporting on certain incidents,such as the 2021 order requiring manufacturers and operators of vehicles equipped with automated driving systems and advanced driver assistance syste

128、ms to report crashes.43 Between 2016 and 2021,the NHTSA investigated 42 crashes that likely involved driving assistance systems.44 Since 2021,however,the NHTSA recorded a total of 522 crashes involving various levels of automated driving systems just from data collected between July 2021 to May 2023

129、.45 The substantial increase in incident reports following the announcement of the 2021 order was likely supported by the mandatory reporting approach.As vehicles with various levels of automated driving systems become more commonly used on public roads,understanding the potential safety issues and

130、trends in automated driving systems can be enhanced by the number of incidents reported and data collected.Apart from issuing orders for specific incident reporting obligations,the NHTSA also provides a citizen reporting portal on their website,where individuals can report safety concerns related to

131、 their vehicle,tires,car seat,or equipment.46 Data collected from this portal helps the NHTSA detect safety issues from vehicle usage,launch investigations on possible defects,and initiate safety recalls when necessary.Also observed in the transportation sector is the Aviation Safety Reporting Syste

132、m,which is a voluntary reporting system emphasizing human performance in the aviation industry.The ASRS receives reports on both unsafe occurrences and hazardous situations,submitted by pilots,air traffic controllers,dispatchers,cabin crew,maintenance technicians,unmanned aircraft systems crew,and o

133、thers.Center for Security and Emerging Technology|26 Cybersecurity:A Shift to Mandatory Reporting In recent years,the U.S.government has begun implementing a succession of new regulations and guidelines to make reporting cyber incidents mandatory in various domains.Previously,cyber incident regulati

134、ons primarily focused on infrastructure resilience and data privacy to manage cyber incidents and keep organizations accountable.There was no widely established federal policy framework mandating reporting of cyber incidents,and the limited emphasis on collecting incident data also meant that learni

135、ng from previous incidents was less of a priority.47 NIST,in collaboration with MITRE,launched the U.S.National Vulnerability Database in 2005,to provide a collection and knowledge base of cybersecurity vulnerability incidents.48 The NVD was developed upon and synchronized with the Common Vulnerabil

136、ity Enumeration(CVE)list,which is a voluntary reporting framework operated by MITRE that was launched publicly in 1999.49 In 2016,NIST released its“Guide to Cyber Threat Information Sharing,”which recommended sharing Cyber Threat Informationthat is,any information that can help an organization ident

137、ify,assess,monitor,and respond to cyber threats,including findings from analyses of incidentsto improve cybersecurity within organizations.50 Surveys showed a growing trend of organizations using CTI,but the use of CTI faced several challenges stemming from the absence of a federal mandate for an in

138、cident reporting policy framework.51 Organizations struggled to find reliable and comprehensive sources of CTI,and it was unclear what information could be shared,how it could be shared,and whether their information-sharing practices were compliant.52 Center for Security and Emerging Technology|27 D

139、efinition of Cyber Incidents Cyber incident.An event occurring on or conducted through a computer network that actually or imminently jeopardizes the integrity,confidentiality,or availability of computers,information or communications systems or networks,physical or virtual infrastructure controlled

140、 by computers or information systems,or information resident thereon.For purposes of this directive,a cyber incident may include a vulnerability in an information system,system security procedures,internal controls,or implementation that a threat source could exploit.Significant cyber incident.A cyb

141、er incident that is(or a group of related cyber incidents that together are)likely to result in demonstrable harm to the national security interests,foreign relations,or economy of the United States,or to the public confidence,civil liberties,or public health and safety of the American people.Source

142、:Presidential Policy Directive 41(PPD-41):United States Cyber Incident Coordination.53 Center for Security and Emerging Technology|28 The rapid emergence of new technologies coupled with the advancement of AI resulted in the proliferation of CyberAI threats that brought a new urgency to the field of

143、 cybersecurity.54 As these technologies become increasingly integral to both public and private sectors,the U.S.government has recognized the existence of the crucial gap in cyber incident reporting and has been actively formulating initiatives to address it.This shift signified a departure from sof

144、t lawssuch as standardsand reflects the U.S.governments commitment to improving its understanding of cyber incidents and bolstering its response and resilience to future threats.These emerging proposals and regulations pertain to various entities,including financial service providers,critical infras

145、tructure providers,and public companies.The Cyber Incident Reporting for Critical Infrastructure Act of 2022(CIRCIA)was signed into law in March 2022,directing the Cybersecurity and Infrastructure Security Agency(CISA)to develop and implement mandatory incident reporting.55 CIRCIA requires providers

146、 of critical infrastructure to report substantial cyber incidents to CISA within 72 hours,while ransomware attacks where payment occurred must be reported within 24 hours.The Federal Information Security Modernization Act of 2014(FISMA)requires federal Executive Branch civilian agencies to alert CIS

147、A on cybersecurity incidents involving their information and information systems.56 FISMA includes guidelines to align incident reporting disclosure information and a one-hour notification timeframe;it also moved root cause analysis to the end of the incident-handling process to allow agencies to no

148、tify response teams sooner.The Securities and Exchange Commission has adopted new requirements for public companies to report and disclose security breaches or incidents.57 The Office of the Comptroller of the CurrencyTreasury,the Board of Governors of the Federal Reserve System,and the Federal Depo

149、sit Insurance Corporation have issued a rule requiring banking organizations to notify their federal regulator of cyber incidents within 36 hours.58 Center for Security and Emerging Technology|29 The National Credit Union Administration Board has approved a final rule requiring federally insured cre

150、dit unions to notify them of cyber incidents within 72 hours.59 Cyber incidents have been around for decades,and it is only recently that federated incident reporting policy frameworks are being established to document them systematically,improve information sharing and situational awareness of inci

151、dents among response teams,and shorten incident response time.The emerging trend of mandatory cyber incident reporting policies could plausibly set a strong foundation and incentive for the early adoption of incident reporting policies in the field of AI.Center for Security and Emerging Technology|3

152、0 Figure 4.Key Takeaways:Incident Reporting From Other Sectors Center for Security and Emerging Technology|31 Discussion Our analysis of the two AI incident reporting databases,emerging government initiatives related to AI incident reporting,and the various incident reporting systems in the healthca

153、re,transportation,and cybersecurity sectors revealed disadvantages and advantages.These insights offered several important lessons that can be applied to an AI incident reporting policy framework,as discussed in the following:Limited incident reporting frameworks are inadequate.Inconsistent data col

154、lection creates meaningless data.There is a need for a federated AI incident reporting framework.Incident investigation supports effective safety policies.Limited Incident Reporting Frameworks Are Inadequate Across the board,the incident reporting initiatives examined in this paper often emphasized

155、either citizen,voluntary,or mandatory reporting,typically focusing on one or two of these reporting categories.In isolation,each of these three frameworks has limitations.Adopting a hybrid framework that incorporates all three reduces the limitations.Our assessment of reporting frameworks in the hea

156、lthcare sector demonstrates that relying on voluntary reporting alone may result in low numbers of reported incidents,and potentially miss incident data.The low numbers may be attributed to the lack of incentive for entities and organizations to report incidents without a reporting obligation.Conseq

157、uently,voluntary reporting is unlikely to be a reliable and sufficient method for capturing an impactful and comprehensive AI incident landscape.The legislative initiatives examined in this paper have embraced mandatory AI incident reporting in their AI policies,underscoring a consensus on the impor

158、tance of collecting and documenting AI incidents.However,limiting incident reporting regulations to mandatory obligations may miss out on incidents that dont fall within regulatory scopes.Supplementing a mandatory reporting framework with voluntary and citizen reporting(similar to those outlined by

159、China)can help identify out-of-scope incidents Center for Security and Emerging Technology|32 and detect novel incidents that emerge during usage.This could be particularly useful as AI systems are usually trained under controlled conditions that do not fully reflect the real-world context in which

160、they are deployed.Employing a hybrid incident reporting framework to collect and document AI incidents will be crucial for documenting a wide array of AI harms and harm dimensions.As AI continues to advance and become more prevalent,AI harms can be expected to grow both in scale and severity.60 The

161、information gathered from incident reports will be essential for policymakers and researchers to gain a more thorough insight into the potential risks associated with AI,and to develop effective safety regulations to reduce the reoccurrence of AI harm.Inconsistent Data Collection Creates Meaningless

162、 Data As AI systems are developed and deployed throughout a wide range of sectors and applications,their impacts will extend across regulatory jurisdictions and geographical boundaries.Relying on state initiatives or domain-specific guidelines will likely not be adequate for aggregating AI incident

163、data that can accurately depict the many dimensions of AI harm.This is evident in our discussion of incident reporting in the healthcare sector,where states have adopted the NQFs Serious Reportable Events differently.The differences have made it difficult to aggregate a national dataset on medical i

164、ncidents to identify healthcare safety trends and systemic issues.Such incongruencies could significantly undermine efforts to identify harmful trends,system vulnerabilities,and the safety measures needed to mitigate risks associated with AI.On the other hand,the mandatory incident reporting policie

165、s in cybersecurity delineated clear instructions for information disclosure,and specific notification timeframes.Clear and intentional guidelines like these may enable timely reporting,improve information sharing,and engender greater data quality and quantity for understanding and mitigating AI inci

166、dents.Furthermore,a standardized disclosure guideline would greatly assist the development of a robust taxonomy and classification framework on AI harms that can enhance information sharing and research on AI safety by enabling comparable data points for analysis.The definitions and classification o

167、f AI harms will be foundational when developing an AI incident reporting framework to Center for Security and Emerging Technology|33 accurately capture the data that will promote our understanding of the various dimensions of emerging AI harms and risks.There Is a Need for a Federated AI Incident Re

168、porting Framework Implementing a federated framework for AI incident reporting is essential as AI is developed and deployed across sectors and applications.A federated approach provides a centralized framework prescribed by a singular authoritative government body or the federal government.The frame

169、work stipulates a set of minimum requirements that can be adapted and implemented across government agencies and non-governmental organizations.A federated AI incident reporting framework can promote comprehensive and consistent collecting,documenting,and sharing of AI incident data.Conversely,relyi

170、ng on individual regulatory agencies or sector-specific frameworks could result in fragmented efforts and inconsistent data.Legislative initiatives from China,the European Union,Brazil,and Canada suggest a growing consensus on reporting AI incidents to mitigate rising concerns about AI harms.The U.S

171、.government has not yet announced significant legislative initiatives outlining a federated AI incident reporting policy framework that includes reporting to external oversight entities.Presently,the U.S.approach to AI incident reporting is generally limited to voluntary and citizen reporting,and ma

172、intains its strategy of directing government organizations to regulate AI incidents within their domains.This could increase the risk of engendering fragmented incident reporting frameworks,such as those observed in the healthcare sector.The value and emphasis different authorities will place on est

173、ablishing an AI incident reporting framework and database will likely vary.This diversity can impact data collection in each domain,making it difficultif not impossibleto aggregate,analyze,and understand trends in AI incidents across sectors.As a result,developing comprehensive measures to mitigate

174、AI harms becomes more challenging.In the healthcare sector,the absence of a federated AI incident reporting policy framework impacted incident data collection efforts.Incident reporting initiatives were fragmented and inconsistent,making it difficult to identify comparable data points for analysis.I

175、n contrast,the transportation sector and the cybersecurity sector have clear policies and standardized rules for reporting incidents.The NTSB and NHTSA have Center for Security and Emerging Technology|34 established incident reporting and investigative mechanisms,facilitating a robust system for doc

176、umenting incidents,identifying root causes,and developing evidence-based safety policies.In the cybersecurity sector,until recently there were uncertainties about how incident information could be shared safely and compliantly,which hindered incident response efforts.However,this will be changing as

177、 the U.S.government has now made incident reporting in cybersecurity mandatory.This move demonstrates the U.S.governments commitment to and endorsement of collecting,documenting,and sharing data on cybersecurity incidents.Incident Investigation Supports Effective Safety Policies A safety investigati

178、on board or research team has been advantageous in identifying root causes of transportation incidents.This has led to the implementation of evidence-based,life-saving measures and safety regulations in the U.S.transportation sector.The NTSBs and NHTSAs approach to using incident investigation facil

179、itates a direct link between data collection and policy responses,leveraging in-depth investigations to support informed decision-making.An investigative safety board would be equally useful for conducting root-cause analysis of significant AI incidents and providing feedback to help AI actors impro

180、ve their design and development,and enable policymakers to craft effective regulations and educate the public on AI safety.A safety investigation board or safety research team could contribute valuable technical and contextual data to understanding AI harm.Center for Security and Emerging Technology

181、|35 Recommendations Policies promoting the establishment of a federated AI incident reporting framework would ensure a more comprehensive collection of AI incidents data and facilitate the development of an authoritative classification system for extracting meaningful data and trends on AI harm.This

182、 data would support in-depth research on AI safety and system vulnerabilities,enhance our ability to understand potential AI risks,and equally important,help policymakers develop more effective safety regulations and practices to mitigate AI harm.Based on the observations discussed above and the nat

183、ure of AI as a general-purpose technology,we make the following recommendations to address the current gap in AI incident reporting.Establish clear policies for federated hybrid AI incident reporting.Develop a standardized and authoritative classification system.Create an independent AI incident inv

184、estigation agency.Explore automated data collection mechanisms.Establish Clear Policies for Federated Hybrid AI Incident Reporting Policymakers should establish a federated AI incident reporting policy framework to gather incident data across sectors and applications,involving a hybrid of mandatory,

185、voluntary,and citizen reporting.It should include clear guidelines on implementing a hybrid of mandatory,voluntary,and citizen reporting policies.Reporting should be made to an independent external committee(government agency,professional association,oversight body)to promote transparency and accoun

186、tability in AI incident management.The incident reporting policy framework should be incorporated into national legislative AI proposal packages to ensure a comprehensive implementation across sectors and applications.Center for Security and Emerging Technology|36 Mandatory Reporting Relevant AI act

187、ors should be mandated to report covered incidents promptly.The rise in regulatory mandates for reporting cyber incidents signals the U.S.governments commitment to enhancing resilience and safety in cyber technology.Policymakers should leverage this shift and advocate for similar support in implemen

188、ting mandatory incident reporting in AI.Mandatory reporting can promote consistent AI incident reporting,prevent data gaps across sectors and applications,and provide a comprehensive knowledge base on AI harm that can inform research on AI safety and risks.An in-depth assessment will be necessary to

189、 define covered incidents involving the types of harm,scale,and AI actors to make the reporting obligation proportionate.Voluntary Reporting Voluntary reporting frameworks should also be established alongside the mandatory framework to capture AI incidents outside the mandatory jurisdiction.AI actor

190、s should be permitted and encouraged to report AI incidents that fall outside regulatory scopes voluntarily,usually to a government agency or professional groups.This would have lesser compliance obligations on AI actors compared to mandatory reporting.Though voluntary,the data collected from volunt

191、ary reporting can enhance the overall data fidelity of documented AI incidents.Supplementing the mandatory reporting framework with a voluntary option may also reduce resistance to implementing a mandatory framework.Citizen Reporting Similarly,aligned with the values of democratic governance,an easi

192、ly accessible reporting framework should be made available for citizen reporting to document AI incidents.While AI system providers and operators should be required to report AI incidents,other stakeholders and the public should also be able to report AI harm they may have experienced.When designing

193、 a citizen reporting system,special attention should be given to vulnerable populations and underrepresented communitiesgroups disproportionately affected by biased AI systems.Relevant stakeholders should be included meaningfully in the development process of the reporting system to ensure their nee

194、ds and concerns are adequately addressed and incorporated.Center for Security and Emerging Technology|37 Develop a Standardized and Authoritative Classification System The AI incident reporting framework should include a standardized set of disclosed information plus accommodations for the unique ch

195、aracteristics of distinct domains,such as privacy concerns and other regulatory requirements.Standardizing the disclosure system can promote greater consistency in the collected data,allow comparable analyses,and reduce the risk of missing crucial information from incidents across different domains.

196、Implementing a standardized disclosure system can also contribute to developing a robust classification framework on AI harm,providing a common foundation for identifying AI harm and thus enhancing our analysis on the subject.Create an Independent AI Incident Investigation Agency When a significant

197、AI incident occurs,an independent board should investigate the root cause and objectively analyze the incident.*This will provide extra scrutiny over significant AI incidents,keep AI actors accountable,and retrieve valuable technical and contextual data about the incidents.AI actors should be compel

198、led to design AI systems with mechanisms supporting investigations and data collection.Furthermore,establishing an investigative agency will help ensure appropriate response measures for significant AI incidents.This agency will play a vital role in addressing and mitigating adverse consequences res

199、ulting from AI use.Outcomes from these investigations will provide key insights into significant incidents,enabling the agency to recommend safety regulations to reduce the risk of similar incidents from reoccurring.Explore Automated Data Collection Mechanisms Automated data collection mechanismssuc

200、h as flight recorderscan provide crucial technical and contextual information that facilitates root-cause analysis of accidents,one of the methods used by the NTSB to collect information on transportation *The definition of a significant AI incident should be determined during the development of a s

201、tandardized taxonomy framework.Center for Security and Emerging Technology|38 incidents.Comparable mechanisms for AI systems should be explored.Obtaining technical and contextual information from AI incidents would be highly advantageous.For example,such a mechanism could capture critical informatio

202、n about the systems environment,or a“snapshot”of the models technical data during an incident.This information could address concerns pertinent to the issues of explainability in AI harms.At the same time,automated data collection mechanisms in AI systems could raise concerns about proprietary data

203、and security issues.These issues should be addressed thoroughly to avoid pushback from companies.Additionally,automated data-collection mechanisms do not replace other incident-reporting systems.Rather,they supplement the data collected with additional technical and contextual information.Center for

204、 Security and Emerging Technology|39 Conclusion The present moment offers a prime opportunity to establish an AI incident reporting framework with relatively low stakes.However,this window is rapidly closing as AI becomes more prevalent across applications and sectors.A federated,comprehensive,and s

205、tandardized framework will prevent data gaps and enhance data quality.Adopting a hybrid framework that includes mandatory,voluntary,and citizen reporting will improve data fidelity,providing a more accurate representation of the emerging trends in AI harm and risk.Further research will be necessary

206、to determine the details for operationalizing such an AI incident reporting policy framework.An AI incident reporting framework must be integrated as an essential component of AI safety rather than developed as an afterthought in AI legislative initiatives.Clear obligations and disclosure requiremen

207、ts should be outlined from the outset to enable frictionless compliance from relevant AI actors.Likewise,easily accessible and comprehensive reporting platforms should be made available to the public so the database may capture novel,unexpected incidents that may emerge during usage.Lessons from the

208、 healthcare,transportation,and cybersecurity sectors provided a compelling argument for implementing a federated mandatory incident reporting system that will positively affect safety practices.In places where mandatory incident reporting has been implemented,evidence of higher reporting rates has b

209、een observed and associated with a more positive safety culture and a significant reduction in adverse events.61 Being at the early stages of AI harm research,the data gathered from a comprehensive and systematic incident reporting system would greatly assist and expedite our knowledge in this area.

210、Policymakers will be better equipped to propose more precise and effective safety regulations,and researchers will gain greater clarity on both the short-and long-term risks associated with AI.The ability to mitigate AI harms and manage their aftermath competently can shape public conversations abou

211、t AI usage.Nuclear plant disasters such as Chernobyl,Three Mile Island,and Fukushima have had adverse effects on global perceptions of nuclear energy.62 In the aftermath of these instances,public opinions shifted on nuclear energy,and governments either significantly delayed implementation plans or

212、reinforced their stance against nuclear power.Even the German parliament,which has long stood by Center for Security and Emerging Technology|40 technologically safe nuclear power plants,voted to phase out nuclear power plants shortly after the Fukushima disaster in 2011.63 The country closed its las

213、t nuclear power plants in 2023.64 Presently,a growing percentage of Americans say they feel more concerned than excited about the increased use of artificial intelligence,rising from 38 percent in 2022 to 52 percent in 2023.65 A comprehensive incident reporting system can help mitigate these fears b

214、y providing valuable insights that can inform effective safety measures,leading to enhanced AI safety and promoting public trust in the technology.As more data becomes available from AI incident reporting,improving our understanding of AI harms and risks,the policies for the incident reporting frame

215、work should be assessed regularly to determine its robustness and capacity for recording and tracking AI incidents.Such iterative practices should be applied to most AI governance initiatives,as there are still uncertainties surrounding emerging technologies and their impact on society.Center for Se

216、curity and Emerging Technology|41 Authors Ren Bin Lee Dixon is an AI policy analyst researching AI policies,governance,and ethics.Heather Frase,PhD,is a senior fellow at CSET and leads the AI Assessment line of research.Dr.Frase serves on the board of the Responsible AI Collaboration(TheCollab),an o

217、rganization chartered to advance the AI Incident Database and providing editorial oversight for it.Acknowledgements For their comprehensive and valuable reviews,we would like to thank Sean McGregor,Violet Turri,Borhane Blili-Hamelin,Mia Hoffman,Mina Narayanan,Josh Goldstein,Helen Toner,and Zach Arno

218、ld.Finally,we would like to thank Christian Schoeberl and Jason Ly for their assistance in providing and designing the figures and tables,and Margarita Konaev and Igor Mikolic-Torreira for their feedback and support.2024 by the Center for Security and Emerging Technology.This work is licensed under

219、a Creative Commons Attribution-Non Commercial 4.0 International License.To view a copy of this license,visit https:/creativecommons.org/licenses/by-nc/4.0/.Document Identifier:doi:10.51593/20230046 Center for Security and Emerging Technology|42 Endnotes 1 Ewen Callaway,“The entire protein universe:A

220、I predicts shape of nearly every known protein,”Nature 608,no.7921(July 29,2022):1516,https:/doi.org/10.1038/d41586-022-02083-2;Gary Liu et al.,“Deep learning-guided discovery of an antibiotic targeting Acinetobacter baumannii,”Nature Chemical Biology,May 25,2023,19,https:/doi.org/10.1038/s41589-023

221、-01349-8.2 Jonas Degrave et al.,“Magnetic Control of Tokamak Plasmas through Deep Reinforcement Learning,”Nature 602,no.7897(February 2022):41419,https:/doi.org/10.1038/s41586-021-04301-9.3 Mia Hoffman and Heather Frase,“Adding Structure to AI Harm:An Introduction to CSETs AI Harm Framework,”Center

222、for Security and Emerging Technology(July 2023),16,https:/cset.georgetown.edu/publication/adding-structure-to-ai-harm/.4 Julia Angwin et al.,“Machine Bias,”ProPublica,May 23,2016,https:/www.propublica.org/article/machine-bias-risk-assessments-in-criminal-sentencing;Larry Hardesty,“Study finds gender

223、 and skin-type bias in commercial artificial-intelligence systems,”MIT News|Massachusetts Institute of Technology,February 11,2018,https:/news.mit.edu/2018/study-finds-gender-skin-type-bias-artificial-intelligence-systems-0212;“Amazon Scraps a Secret A.I.Recruiting Tool That Showed Bias against Wome

224、n,”CNBC,October 10,2018,https:/ Davis,“AI skin cancer diagnoses risk being less accurate for dark skin study,”The Guardian,November 9,2021,sec.Society,https:/ Heikkil,“Dutch scandal serves as a warning for Europe over risks of using algorithms,”POLITICO(blog),March 29,2022,https:/www.politico.eu/art

225、icle/dutch-scandal-serves-as-a-warning-for-europe-over-risks-of-using-algorithms/.5“Incident 545:Chatbot Tessa gives unauthorized diet advice to users seeking help for eating disorders,”AI Incident Database,accessed November 21,2023,https:/incidentdatabase.ai/cite/545/.6“Incident 543:Deepfake of Exp

226、losion Near US Military Administration Building Reportedly Causes Stock Dip,”AI Incident Database,accessed November 21,2023,https:/incidentdatabase.ai/cite/543/.7“Incident 550:Tesla Allegedly on Autopilot Struck High School Student Exiting School Bus,”AI Incident Database,accessed November 21,2023,h

227、ttps:/incidentdatabase.ai/cite/550/.8“Welcome to the AI Incident Database,”AI Incident Database,accessed July 19,2023,https:/incidentdatabase.ai/.Center for Security and Emerging Technology|43 9“AIAAIC,”AI,algorithmic,and automation incidents and controversies,accessed July 19,2023,https:/www.aiaaic

228、.org/home.10“AVID,”AI Vulnerability Database(AVID),accessed July 19,2023,https:/avidml.org/.11“AI Litigation Database,”Ethical Tech Initiative,accessed July 19,2023,https:/blogs.gwu.edu/law-eti/ai-litigation-database/.12 Carol Anderson et al.,“Response from the AI Risk and Vulnerability Alliance to

229、the NTIA AI Accountability Policy Request for Comment,”AI Risk and Vulnerability Alliance(ARVA)2023,https:/ See AIAAIC.14“AIAAIC Repository,”accessed September 19,2023,https:/ Responsible AI Collective,“Founding Report,”March 28,2022,https:/ the Responsible AI Collaborative Founding Staff,”AI Incide

230、nt Database,accessed November 21,2023,https:/incidentdatabase.ai/blog/join-raic/.16 See AIID.17“What is the GMF Taxonomy?,”Artificial Intelligence Incident Database,accessed July 19,2023,https:/incidentdatabase.ai/taxonomy/gmf/.Data from CSET,“CSETs Harm Taxonomy for the AI Incident Database,”GitHub

231、,accessed November 1,2023,https:/ China Law Translate,“Provisions on the Management of Algorithmic Recommendations in Internet Information Services,”China Law Translate(blog),January 4,2022,https:/ Law Translate,“Provisions on the Administration of Deep Synthesis Internet Information Services,”China

232、 Law Translate(blog),December 12,2022,https:/ China Law Translate,“Interim Measures for the Management of Generative Artificial Intelligence Services,”China Law Translate(blog),July 13,2023,https:/ for Security and Emerging Technology|44 20 European Commission,“Proposal for Laying Down Harmonised Ru

233、les on Artificial Intelligence,”2021,https:/eur-lex.europa.eu/legal-content/EN/TXT/HTML/?uri=CELEX:52021PC0206.21 Senador Rodrigo Pacheco(PSD/MG),“Projeto de Lei N 2338,de 2023”(2023),https:/legis.senado.leg.br/sdleg-getter/documento?dm=9347593&ts=1683152235237&disposition=inline&_gl=1*edqnkm*_ga*MT

234、gyMDY0MTcwMS4xNjc5OTM2MTI0*_ga_CW3ZH25XMK*MTY4MzIxNzUzMy4yLjEuMTY4MzIyMDAyMy4wLjAuMA.22 Minister of Innovation,Science and IndustryCanada,“Bill C-27,An Act to enact the Consumer Privacy Protection Act,the Personal Information and Data Protection Tribunal Act and the Artificial Intelligence and Data

235、Act and to make consequential and related amendments to Other Acts”(2022),https:/www.parl.ca/DocumentViewer/en/44-1/bill/C-27/first-reading.23 Exec.Order No.14110,Safe,Secure,and Trustworthy Development and Use of Artificial Intelligence,”202324283 88 FR 75191(November 1,2023),https:/www.federalregi

236、ster.gov/documents/2023/11/01/2023-24283/safe-secure-and-trustworthy-development-and-use-of-artificial-intelligence.24 NAIAC,“RECOMMENDATION:Improve Monitoring of Emerging Risks from AI through Adverse Event Reporting”(National Artificial Intelligence Advisory Committee,November 2023),https:/ai.gov/

237、wp-content/uploads/2023/12/Recommendation_Improve-Monitoring-of-Emerging-Risks-from-AI-through-Adverse-Event-Reporting.pdf.25 National Institute of Standards and Technology,“Artificial Intelligence Risk Management Framework:AI RMF(1.0),”Gaithersburg,MD:National Institute of Standards and Technology(

238、January 2023),https:/doi.org/10.6028/NIST.AI.100-1.26 The White House,“FACT SHEET:Biden-Harris Administration Secures Voluntary Commitments from Leading Artificial Intelligence Companies to Manage the Risks Posed by AI,”The White House,July 21,2023,https:/www.whitehouse.gov/briefing-room/statements-

239、releases/2023/07/21/fact-sheet-biden-harris-administration-secures-voluntary-commitments-from-leading-artificial-intelligence-companies-to-manage-the-risks-posed-by-ai/.27 Institute of Medicine(US)Committee on Quality of Health Care in America;Kohn LT,Corrigan JM,Donaldson MS,editors.To Err is Human

240、:Building a Safer Health System.Washington(DC):National Center for Security and Emerging Technology|45 Academies Press(US);2000.https:/www.ncbi.nlm.nih.gov/books/NBK225187/2,Errors in Health Care:A Leading Cause of Death and Injury.28“HAI and Antibiotic Use Prevalence Survey,”Centers for Disease Con

241、trol and Prevention,March 31,2022,https:/www.cdc.gov/hai/eip/antibiotic-use.html;Craig Umscheid et al.,“Estimating the Proportion of Healthcare-Associated Infections That Are Reasonably Preventable and the Related Mortality and Costs,”Infection Control and Hospital Epidemiology 32,no.2(February 2011

242、),https:/doi.org/10.1086/657912.29“Serious Reportable Events aka Never Events,”National Quality Forum(NQF),accessed August 15,2023,https:/www.qualityforum.org/Topics/SREs/Serious_Reportable_Events.aspx.30“Home|Patient Safety Organization(PSO)Program,”Agency for Healthcare Research and Quality(AHRQ),

243、accessed August 23,2023,https:/pso.ahrq.gov/.31“Sentinel Event Policy and Procedures|The Joint Commission,”The Joint Commission,accessed August 16,2023,https:/www.jointcommission.org/resources/sentinel-event/sentinel-event-policy-and-procedures/.32“State-Based Reporting in Healthcare,”National Quali

244、ty Forum(NQF),accessed July 23,2023,https:/www.qualityforum.org/Projects/State_Based_Reporting/State-Based_Reporting_in_Healthcare.aspx.33 National Quality Forum(NQF),“Variability of State Reporting of Adverse Events,”October 2011.https:/www.qualityforum.org/Topics/SREs/State_Variability_Fact_Sheet.

245、aspx 34 Agency for Healthcare Research and Quality,“Network of Patient Safety Databases Chartbook,2021”(Rockville,MD:AHRQ,August 2021).https:/www.ahrq.gov/sites/default/files/wysiwyg/npsd/data/npsd-chartbook-2021.pdf 35“Listed PSOs,”Agency for Healthcare Research and Quality(AHRQ),accessed August 16

246、,2023,https:/pso.ahrq.gov/pso/listed;AHA Hospital Statistics,“Fast Facts on U.S.Hospitals,2023”(AHA Hospital Statistics),accessed September 10,2023,https:/www.aha.org/system/files/media/file/2023/05/Fast-Facts-on-US-Hospitals-2023.pdf.36 Agency for Healthcare Research and Quality,“Network of Patient

247、 Safety Databases Chartbook,2021,”1.37 Ellen Flink et al.,“Lessons Learned from the Evolution of Mandatory Adverse Event Reporting Systems,”in Advances in Patient Safety:From Research to Implementation(Volume 3:Implementation Center for Security and Emerging Technology|46 Issues),ed.Kerm Henriksen e

248、t al.,Advances in Patient Safety(Rockville,MD):Agency for Healthcare Research and Quality(US),2005),http:/www.ncbi.nlm.nih.gov/books/NBK20547/.38 Sentinel Event Policy and Procedures.The Joint Commission.https:/www.jointcommission.org/resources/sentinel-event/sentinel-event-policy-and-procedures/.39

249、 Flink et al.,“Lessons Learned from the Evolution of Mandatory Adverse Event Reporting Systems.”40 The National Transportation Safety Board,accessed July 14,2023.https:/www.ntsb.gov/Pages/home.aspx.41“20212023 Most Wanted List,”National Transportation Safety Board(NTSB),accessed February 24,2024,htt

250、ps:/www.ntsb.gov/Advocacy/Pages/ArchiveMWL.aspx.42“Home|NHTSA,”Text,National Highway Traffic Safety Administration,accessed October 21,2023,https:/www.nhtsa.gov/.43 NHTSA,“Standing General Order 2021-01|Incident Reporting for Automated Driving Systems(ADS)and Level 2 Advanced Driver Assistance Syste

251、ms(ADAS),”Text,National Highway Traffic Safety Administration,2021,https:/www.nhtsa.gov/document/sgo-crash-reporting-adas-ads.44 Faiz Siddiqui,Rachel Lerman,and Jeremy B.Merrill,“Teslas running Autopilot involved in 273 crashes reported since last year,”The Washington Post,June 15,2022,https:/ NHTSA

252、,“Summary Report:Standing General Order on Crash Reporting for Level 2 Advanced Driver Assistance Systems,”National Highway Traffic Safety Administration(June 2022),https:/www.nhtsa.gov/sites/nhtsa.gov/files/2022-06/ADAS-L2-SGO-Report-June-2022.pdf;NHTSA,“Summary Report:Standing General Order on Cra

253、sh Reporting for Automated Driving Systems,”National Highway Traffic Safety Administration(June 2022),https:/www.nhtsa.gov/sites/nhtsa.gov/files/2022-06/ADS-SGO-Report-June-2022.pdf.46“Report a Safety Problem|NHTSA,”NHTSA,accessed September 15,2023,https:/www.nhtsa.gov/report-a-safety-problem.47 Cla

254、re M.Patterson,Jason R.C.Nurse,and Virginia N.L.Franqueira,“Learning from cyber security incidents:A systematic review and future research agenda,”Computers&Security 132(September 1,2023):103309,https:/doi.org/10.1016/j.cose.2023.103309.Center for Security and Emerging Technology|47 48 CVE,“Related

255、Efforts|CVE,”CVE,accessed November 29,2023,https:/www.cve.org/About/RelatedEfforts.49 CVE,“History|CVE,”CVE,accessed November 29,2023,https:/www.cve.org/About/History.50 Chris Johnson et al.,“Guide to Cyber Threat Information Sharing,”National Institute of Standards and Technology Special Publicatio

256、n(October 2016),https:/doi.org/10.6028/NIST.SP.800-150.51 Rebekah Brown and Robert M Lee,“The Evolution of Cyber Threat Intelligence(CTI):2019 SANS CTI Survey,”2019,https:/a51.nl/sites/default/files/pdf/Survey_CTI-2019_IntSights.pdf 52 Konstantinos Rantos et al.,“Interoperability Challenges in the C

257、ybersecurity Information Sharing Ecosystem,”Computers 9,no.1(March 2020):18,https:/doi.org/10.3390/computers9010018.53“Presidential Policy Directive-United States Cyber Incident Coordination,”Obama White House,July 26,2016,https:/obamawhitehouse.archives.gov/the-press-office/2016/07/26/presidential-

258、policy-directive-united-states-cyber-incident.54 Ben Buchanan et al.,“Automating Cyber Attacks,”Center for Security and Emerging Technology(November 2020),https:/doi.org/10.51593/2020CA002.55 CISA,“Cyber Incident Reporting for Critical Infrastructure Act of 2022(CIRCIA)Fact Sheet,”Cybersecurity and

259、Infrastructure Security Agency(CISA),(2023),https:/www.cisa.gov/sites/default/files/2023-01/CIRCIA_07.21.2022_Factsheet_FINAL_508%20c.pdf.56“Federal Incident Notification Guidelines,”Cybersecurity&Infrastructure Security Agency(CISA),accessed July 6,2023,https:/www.cisa.gov/federal-incident-notifica

260、tion-guidelines.57“Cybersecurity Risk Management,Strategy,Governance,and Incident Disclosure,”U.S.Securities and Exchange Commission,November 14,2023,https:/www.sec.gov/corpfin/secg-cybersecurity.58 Federal Register,“Computer-Security Incident Notification Requirements for Banking Organizations and

261、Their Bank Service Providers,”12 CFR 53 12 CFR 225 12 CFR 304 (November 23,2021),https:/www.federalregister.gov/documents/2021/11/23/2021-25510/computer-security-incident-notification-requirements-for-banking-organizations-and-their-bank.59“Cyber Incident Notification Requirements,”National Credit U

262、nion Administration(NCUA),August 14,2023,https:/ncua.gov/regulation-supervision/letters-credit-unions-other-guidance/cyber-incident-notification-requirements.Center for Security and Emerging Technology|48 60 Nestor Maslej et al.,“Artificial Intelligence Index Report 2023,”AI Index Steering Committee

263、(Stanford,CA:Institute for Human-Centered AI,Stanford University,April 2023),https:/aiindex.stanford.edu/wp-content/uploads/2023/04/HAI_AI-Index-Report_2023.pdf.61 A.Hutchinson et al.,“Trends in healthcare incident reporting and relationship to safety and quality data in acute hospitals:results from

264、 the National Reporting and Learning System,”BMJ Quality&Safety 18,no.1(February 1,2009):510,https:/doi.org/10.1136/qshc.2007.022400;Elena Ramrez et al.,“Effectiveness and limitations of an incident-reporting system analyzed by local clinical safety leaders in a tertiary hospital,”Medicine 97,no.38(

265、September 21,2018):e12509,https:/doi.org/10.1097/MD.0000000000012509.62 Ortwin Renn,“Public responses to the Chernobyl accident,”Journal of Environmental Psychology 10,no.2(June 1990):15167,https:/doi.org/10.1016/S0272-4944(05)80125-2.63“Die Beschlsse des Bundestages am 30.Juni und 1.Juli,”Deutscher

266、 Bundestag,June 2011,https:/www.bundestag.de/webarchiv/textarchiv/2011/34915890_kw26_angenommen_abgelehnt-205788.64 Catherine Clifford,“Germany has shut down its last three nuclear power plants,and some climate scientists are aghast,”CNBC,April 18,2023,https:/ Alec Tyson and Emma Kikuchi,“Growing public concern about the role of artificial intelligence in daily life,”Pew Research Center(blog),August 28,2023,https:/www.pewresearch.org/short-reads/2023/08/28/growing-public-concern-about-the-role-of-artificial-intelligence-in-daily-life/.