人工智能內部審計提示工程演示.pdf

《人工智能內部審計提示工程演示.pdf》由會員分享，可在線閱讀，更多相關《人工智能內部審計提示工程演示.pdf（30頁珍藏版）》請在三個皮匠報告上搜索。

1、ARTIFICIAL INTELLIGENCE&PROMPT ENGINEERING FOR INTERNAL AUDITAugust 2024AI Standards,Regulations,&Governance03AI and Internal Audit09Prompt Engineering Tips&Tricks15Prompt Patterns&Examples20QuestionsAgendaAgenda2AI Standards,Regulations&GovernanceAI Standards,Regulations&GovernancePoll question 1Po

2、ll question 1What is your primary concern with the use of AI at your organization?Data privacy and confidentiality.AReliability and accuracy of outputs.BPotential biases and fairness issues.CCompliance with relevant laws and regulations.D4Why is AI being regulated?Why is AI being regulated?New risks

3、 and threats for individuals fundamental rights and freedomsNeed to build trust to keep the momentum in use and adoption of AIDisruptive changes on business,economy,social life,environment,science and researchAbility to enable manipulation and advanced surveillanceSelf-learning and self-evolving nat

4、ure following the launchAI-specific Frameworks and Regulationsi.e.,EU AI Act,NIST AI Risk Management FrameworkSectorial Regulations aligned to AIi.e.,UK Consumer DutyRegulations on Data and Digital Infrastructurei.e.,DORA(Digital Operational Resilience Act),GDPR5Recognized standards for AI Governanc

5、eRecognized standards for AI GovernanceNIST AI Risk Management Framework is designed to equip organizations and individuals with approaches that increase the trustworthiness of AI systems,and to help foster the responsible design,development,deployment,and use of AI systems over time.European Union

6、AI Act focuses primarily on strengthening rules around data quality,transparency,human oversight and accountability.It also aims to address ethical questions and implementation challenges.ISO/IEC AI Framework provides guidance on managing risk associated with the development and use of AI.The docume

7、nt offers strategic guidance to organizations to assist in integrating risk management into significant activities and functions.ISO/IEC 23894:2023EU AI ActNIST AI100-1Predominant PrinciplesAccountabilityHuman-Centered DesignTransparency and ExplainabilityFairness and Bias MitigationPrivacy and Data

8、 GovernanceRobustness and Security6Principles of Responsible AI*Principles of Responsible AI*Organisations should strive for transparency in their AI systems,making sure they are understandable and explainable.Clear explanations of how AI systems work,including data sources,algorithms,and decision-m

9、aking processes,foster trust and enable users to make informed judgments.To ensure fairness,organisations should proactively identify and mitigate biases in their AI systems.They should assess and address potential biases in data collection,model training,and algorithmic decision-making to prevent u

10、nfair outcomes or discrimination based on protected attributes such as race,gender,or ethnicity.Organisations should establish mechanisms to ensure accountability for the development,deployment,and use of AI systems.This includes clear lines of responsibility,monitoring and auditing processes,and me

11、chanisms for addressing potential harms or unintended consequences of AI applications.Safeguarding user privacy and ensuring responsible data governance is crucial.Organisations must handle personal and sensitive data with utmost care,following legal and ethical guidelines.They should implement appr

12、opriate measures to protect data,obtain informed consent,and comply with relevant data protection regulations.AI systems should be designed and developed to be robust,resilient,and secure.Organisations must consider potential vulnerabilities,adversarial attacks,and potential risks associated with AI

13、 deployment.Implementing strong security measures,continuous monitoring,and risk assessment processes are essential to maintain system integrity.Organisations should prioritise human well-being and ensure that AI systems are designed with human needs and values in mind.This involves involving divers

14、e perspectives,considering user feedback,and regularly assessing the social impact and ethical implications of AI applications.Transparency and ExplainabilityFairness and Bias MitigationAccountabilityPrivacy and Data GovernanceRobustness and SecurityHuman-Centered Design123456*Common themes in NIST

15、AI RMF Characteristics of Trustworthy AI,Microsoft Responsible AI Standard,AWS Core Dimensions of Responsible AI,OECD AI Principles,and others mentioned.7Five key steps to get started with AI governance&IA ConsiderationsFive key steps to get started with AI governance&IA Considerations8GovernancePol

16、icy,Standards,and ControlsTrainingAI Inventory&Intake ProcessRisk-Prioritized ApproachEnsures cross-functional representation and leadership buy-in,fostering cohesive decision-making and accountability in AI governance initiativesProvide guidelines and standards to employees for AI development and d

17、eployment,ensuring alignment with ethical principles,legal requirements,and organizational valuesHelps employees understand AI technologies,their implications,and best practices for effective governance,enhancing the organizations capacity to responsibly manage AI initiativesEnables organizations to

18、 gain awareness of different AI initiatives in flight,AI platforms and capabilities(including third parties),facilitating risk assessment,compliance monitoring,and resource allocation for governance effortsAllows organizations to allocate resources effectively,focusing on addressing the most critica

19、l concerns first and mitigating potential harms associated with AI deployments12345Explore participation in AI Governance steering committees or working groups to drive awareness for audit purposes,provide risk and control guidance,and assess the impact of AI implementation on audit plans and the br

20、oader audit universeEngage in advisory and consultancy roles early and often to evaluate the design of AI program implementation,helping to avoid gaps and limit risk exposureUnderstand the risk assessment process and consider the businesss use of AI when planning future audit assessmentsInternal Aud

21、it ConsiderationsAI and Internal AuditAI and Internal AuditPoll question 2Poll question 2How would you describe your internal audit departments current use of AI including Generative AI and Machine Learning?We do not use AI in our internal audit processes.AWe are currently exploring AI options for i

22、nternal audit.BWe use AI for specific tasks within the audit process,such as planning or reporting.CAI is fully integrated into our internal audit processes,from planning to reporting.D1011How can Generative AI be applied in Internal Audit?How can Generative AI be applied in Internal Audit?Audit Ann

23、ouncementSelect your departmentInternal AuditPlanning&Scoping MemoDocument Request ListWork Program DevelopmentPlanningFieldworkBusinessProcess FlowsProcess/Control NarrativesEvidence Review AccelerationWork Program CompletionExecutive Summary DevelopmentObservations&RecommendationsAudit Response An

24、alyzerInternal Audit Charter CreationAudit Policies&ProceduresAudit Analytics&AutomationMeeting Notes CleanerAudit Strategy&RoadmapStakeholder CommunicationFunction ResearchControl Test Plan CompletionControl Test Plan DevelopmentBest Practices&Value AddsInnovation BrainstormingRisk AssessmentOtherT

25、odayImpactful Presentation IdeasPrevious 7 DaysAudit Committee PreparationCombative Control OwnerPlanning API AuditMentoring Audit StaffPrevious 30 DaysPreparing for IIA ConferencePolicy&Standard ReviewerLLMs can drive efficiencies throughout the entire audit life cycle.Below are example use-cases f

26、or consideration.12Enabling process walkthroughs using Copilot in MS TeamsEnabling process walkthroughs using Copilot in MS TeamsLeverage Copilot to not only capture and summarize meeting notes,but also steer discussions in real-timeAI-enabled Walkthrough Best PracticesBefore the meeting:Prepare tar

27、geted questions related to the processes,controls,risks,and pain points to be discussedDuring the meeting:Generate follow-up questions based on the discussion to dig deeper into potential issues or areas requiring clarificationReconcile the agenda against the transcript to ensure key points are not

28、missedAfter the meeting:Summarize the discussion,including key action items and follow-up requestsIdentify any potential control gaps or areas for improvement based on the AIs analysis of the meeting contentImportant:Always obtain explicit permission from all participants before recording and be tra

29、nsparent about the intended use of AI to assist with note-taking,analysis,and summarization to ensure trust and compliance with data privacy policies.13Populating the audit notification template using Copilot for MS WordPopulating the audit notification template using Copilot for MS WordIts as simpl

30、e as opening Word,entering a prompt,and generatingHow ProGPT is enabling our teams at ProtivitiHow ProGPT is enabling our teams at Protiviti14Powered by OpenAI ChatGPT 4,ProGPT is a secure generative-AI solution that revolutionizes how we work.This solution allows Protiviti employees to be more effi

31、cient,produce quality results,and further advance our capabilities.We use ProGPT to summarize risk assessments,refine audit report wording,execute research(with internet connectivity),refine audit procedures,and more.The power of secure generative AI at our fingertipsSecureGenerative AIChat with doc

32、umentsPrompt Engineering Tips&TricksPrompt Engineering Tips&Tricks16Level setting on definitions:Level setting on definitions:123Large language models(LLMs):are a type of artificial intelligence(AI)model that can understand and generate text based on the input they receive.LLMs are trained on massiv

33、e datasets containing diverse text samples from various sources,such as books,articles,websites,and more.The training process involves exposing the model to this data repeatedly,allowing it to learn the statistical patterns and linguistic structures inherent in human language.Prompts:are inputs prov

34、ided to a Large Language Model(LLM)to generate responses or perform tasks.They guide the LLM in understanding the context and the type of information or action required.When you give a prompt to an LLM,it processes the text and generates a response based on its training data and the patterns it has

35、learned.Prompt Engineering:involves the strategic design of input prompts to elicit specific responses from large language models(LLMs).This process requires a deep understanding of the models capabilities,training data,and task requirements.The prompts can be structured as text snippets,questions,c

36、ommands,or combinations thereof,tailored to the desired outcome.LLMs also have input and output limitations,so knowing which prompt to use to get the best output is critical.17Elements of a promptElements of a promptContextLogicalExamplesActionableRiskProvide enough background information so the LLM

37、 understands key factors(persona,constraints,area,etc.)Prompts follow a clear line of reasoning to avoid LLM confusion.Offer examples,when possible,to guide the models output.(Shot=input&correlated output)Provide concrete instructions on outputs or actions,making it easier for LLMs to generate usefu

38、l responses.Consider potential risks associated with the output,including hallucination,bias,or other concerns18Limitations to keep in mind:Limitations to keep in mind:Data privacy:As LLMs are trained on massive datasets of text and code they can potentially memorize and leak sensitive information t

39、hat is present in the training data such as personally identifiable information(PII)or Protected Health Information(PHI),trade secrets.To mitigate this risk,robust data anonymization techniques should be used before feeding data into LLMs.Randomness in output:LLM output has a degree of unpredictabil

40、ity or variability in the responses generated by the model.LLMs use complex algorithms and vast training data to generate text,but they also incorporate a degree of randomness to produce diverse and varied outputs making it challenging to rely solely for critical decision-making.To address this limi

41、tation,its essential to validate and cross-reference the output generated by LLMs using human judgment and domain expertise.Dont trust,verify:LLMs work by analyzing the relationships between words in a text corpus and learning to predict the likelihood of a given sequence of words.But this corpus of

42、 text can be out of date,of poor quality&from dubious sources on the internet.To verify the accuracy,human beings must be involved in the benefits and overall sanity of responses from LLMs.Bias:Models trained on unvetted data from the Internet will always mirror human biases;models that constantly l

43、earn from user interaction are also prone to intentional manipulation.To mitigate the risk of discrimination,LLM service providers must carefully evaluate their training datasets for any imbalances that may result in negative consequences.Compliance/regulatory incompatibility:The use of large langua

44、ge models may raise regulatory compliance concerns,such as those related to data privacy,data protection,and intellectual property.This could lead to fines or other penalties for the enterprise that uses the model.Its important to,limit access to sensitive information and adhere to strict data priva

45、cy regulations such as GDPR and HIPAA which is paramount to safeguarding sensitive information.19Building prompt engineering skills will be essential for audit teamsBuilding prompt engineering skills will be essential for audit teamsWhat is Prompt Engineering?Crafting precise,clear,and unambiguous p

46、rompts to elicit accurate and relevant responses from AI systemsGood Prompt:“What are some red flags of financial fraud that auditors should be aware of?”Why is Prompt Engineering important for Auditors?Designing effective questions or statements to guide a conversation with an AI language modelProm

47、otes ethical AI use by reducing biases&ensuring better alignment with human valuesDesigning effective questions or statements to guide a conversation with an AI language modelEnhances the quality and reliability of AI-generated insightsBad Prompt:“Tell me about fraud.”Prompt Patterns&ExamplesPrompt

48、Patterns&Examples21Various Example Prompt TypesVarious Example Prompt TypesThe Persona PatternIn this pattern you will tell the LLM who or what they are to narrow down the training data they draw from and how to best respond to your prompt.Question Refinement PatternIn this pattern you ask the LLM t

49、o generate additional,more refined questions.These questions,in turn,guide the LLM toward crafting superior outputs that align more closely with your desired results.Cognitive Verifier PatternIn this pattern,you are asking the LLM to subdivide a question into additional questions that can be used to

50、 provide a better answer to the original question.This is especially useful when the question being asked is very high level or the user does not have much knowledge about the question.Audience Persona PatternIn this pattern,you will define specific audience details such as geographical location,pro

51、fessional background,personal interests,and other relevant factors to allow the LLM to better customize its output responseFlipped Interaction PatternIn this pattern you will ask the LLM to ask the questions needed to gather information until it can problem-solve the stated goalTemplate PatternIn th

52、is pattern you will tell the LLM to perform a task while giving it a specific outline to follow when generating the output.This prompt saves the user time reworking responses into the format desired.22The Persona PatternThe Persona PatternExample:You are an internal audit consultant:What are some co

53、mmon issue areas that arise when trying to integrate controls after a new system implementationPattern:Act as persona X and perform task YChatGPT:When integrating controls after a new system implementation,several common issue areas can arise.These issues can stem from gaps in understanding,miscommu

54、nication,or inadequate planning.As an internal audit consultant,its important to be aware of these potential issues to help address them effectively.Here are some common problem areas:Lack of Clear Documentation:Incomplete or unclear documentation about the new systems functionality,processes,and co

55、ntrols can lead to confusion and difficulties in integrating appropriate controls.Inadequate Training:Insufficient training for employees on how to use the new system and comply with control requirements can result in errors,non-compliance,and security vulnerabilities.Training programs should cover

56、not only system usage but also control responsibilities and best practices.Incompatibility with Existing Controls:The new system may not align seamlessly with existing control frameworks,leading to gaps or overlaps in control coverage.Its crucial to assess and reconcile any discrepancies to ensure c

57、omprehensive control coverage.Security Vulnerabilities:New systems may introduce security vulnerabilities such as weak access controls,data leakage risks,or integration issues with security tools.Conducting thorough security assessments and implementing necessary safeguards is essentialIn this patte

58、rn you will tell the LLM who or what they are to narrow down the training data they draw from and how to best respond to your prompt.23Question Refinement PatternQuestion Refinement PatternExample:When I ask a question,suggest a better version of the question to use instead.Prompt me if I would like

59、 to use the better version instead:Can you list some accounts payable financial statement risks and their assertions?Pattern:When I ask a question,suggest a better version of the question to use instead (Optional)From now on,whenever I ask a question,suggest a better version of the question to use i

60、nstead (Optional)Prompt me if I would like to use the better version insteadChatGPT:A clearer way to phrase your question might be:Could you provide a list of financial statement risks related to accounts payable,along with their corresponding assertions?Would you like to use this version instead?In

61、 this pattern you ask the LLM to generate additional,more refined questions.These questions,in turn,guide the LLM toward crafting superior outputs that align more closely with your desired results.24Cognitive Verifier Pattern Cognitive Verifier Pattern 1 of 21 of 2In this pattern,you are asking the

62、LLM to subdivide a question into additional questions that can be used to provide a better answer to the original question.This is especially useful when the question being asked is very high level or the user does not have much knowledge about the question.Example:When asked to help design a SOX co

63、mpliance training program,follow these rules.Generate a number of additional questions about the companys budget,SOX requirements,timeline,training objectives,and target audience.Use the insights gathered from these questions to select an appropriate framework(such as COSO,COBIT,or a custom framewor

64、k)that best suits the identified needs.Then,construct a detailed plan for developing and delivering the SOX compliance training program.Pattern:When you are asked a question,follow these rules:Generate a number of additional questions that would help more accurately answer the question Combine the a

65、nswers to the individual questions to produce the final answer to the overall question25Cognitive Verifier Pattern Cognitive Verifier Pattern 2 of 2 2 of 2 ChatGPT:To design an effective SOX compliance training program,we need to start with a deep understanding of your companys specifics and the req

66、uirements set by the Sarbanes-Oxley Act(SOX).Here are some key areas that would require more information:1.Budget:Understanding the budget will help us determine what resources we can allocate,whether it be for employing external consultants,purchasing software solutions or other materials required

67、for the training program.2.SOX Requirements:As SOX is a complex legislation with various sections applicable to different business operations,its essential to clarify which sections are most pertinent to your organization.This will allow us to focus our training on these crucial areas.3.Timeline:Kno

68、wing when you expect the program to be implemented and completed will help in creating a realistic plan that considers preparation time,course duration,and follow-up assessments.4.Training Objectives:Its important we understand what you aim to achieve with this training program beyond compliance.Are

69、 there specific knowledge gaps you want addressed?Do you want to develop a culture of accountability within your staff?5.Target Audience:Are all employees included in this training or only certain departments?The content and delivery method may vary based on the audiences prior knowledge and their r

70、ole within the organization.Example continued from prior slide:26Audience Persona Pattern Audience Persona Pattern In this pattern,you will define specific audience details such as geographical location,professional background,personal interests,and other relevant factors to allow the LLM to better

71、customize its output response.Example:Explain internal controls to me assuming I am a Chief Audit Executive.Pattern:Explain X to me.Assume that I am Persona Y.ChatGPT-Chief Audit ExecutiveThe purpose of internal controls is manifold.Firstly,they aim to maintain the accuracy and completeness of accou

72、nting records,which forms the basis for reliable financial statements.Secondly,they help prevent fraud and reduce the probability of significant errors in accounting records.Moreover,they ensure compliance with applicable laws and regulations such as SOX or GDPR provisions.Regarding their importance

73、 in mitigating risks:Internal controls play a crucial role in identifying potential risks before they become significant issues;this allows management to take timely preventive measures.By ensuring that business operations run smoothly and efficiently internal controls also improve operational effec

74、tiveness.As a CAE your role includes evaluating these internal control systems for their effectiveness verifying that they are properly defined adequately communicated to employees consistently applied and adapted as necessary when conditions change.You would also be responsible for recommending imp

75、rovements if deficits are identified during audits or reviews aligning well with your overall responsibility towards improving operational efficiencies managing risks effectively within your organization while maintaining its financial integrity intact27Flipped Interaction PatternFlipped Interaction

76、 PatternIn this pattern you will ask the LLM to ask the questions needed to gather information until it can problem-solve the stated goal.Example:You are a chief audit executive.I would like you to ask me questions until you have enough information to determine whether a control exception is a contr

77、ol deficiency,significant deficiency,or material weakness.Ask me the first question.See following slidePattern:I would like you to ask me questions to achieve X.You should ask questions until condition Y is met or to achieve this goal(alternatively,forever)(Optional)ask me the questions one at a tim

78、e,two at a time,ask me the first question,etc.28Flipped Interaction Pattern Flipped Interaction Pattern-EXAMPLEEXAMPLEChatGPT:Certainly!To start,could you tell me about the nature of the control exception?Specifically,what is the control designed to achieve,and what was the deviation or issue that o

79、ccurred?User:The control was designed to prevent users from entering and approving their own journal entries.The deviation that occurred was one user had the system access allowing them to enter and approve their own journal entriesChatGPT:Thanks for sharing that information.Now,could you provide so

80、me context about the user who had the system access to enter and approve their own journal entries?For instance,what is their role within the organization,and how critical are their responsibilities related to financial reporting or other key processes?User:Their role was chief accounting officer,th

81、ey are extremely critical to the financial reporting processesChatGPT:Understood.Now,lets delve deeper into the control environment surrounding this situation.Could you describe the overall design of controls related to journal entry approvals in your organization?For example,are there segregation o

82、f duties policies in place,and what other checks and balances exist to prevent unauthorized or inappropriate journal entries?User:Segregation of duties policies are in place,however this user defied the policy.There are no other mitigating controls to prevent unauthorized or inappropriate journal en

83、triesChatGPT:Got it,thank you for clarifying.Now,lets consider the potential impact of this control exception.Could you provide some insights into the financial significance of the journal entries that were entered and approved by the user in question?Specifically,did these entries result in materia

84、l misstatements in the financial statements,or were they relatively minor in terms of their financial impact?User:Material weakness materiality is greater than 4 million impact to the financial statements.The journal entries approved by this user totaled$8 millionChatGPT:Thank you for providing the

85、materiality threshold and the impact of the journal entries approved by the user.One last question:Has there been any evidence of financial misstatements or inaccuracies in the financial statements as a result of these unauthorized journal entries,or were they caught and corrected before the financi

86、al statements were finalized?User:This was not caught before the financial statements were finalized but a post mortem investigation concluded the user did not post any inappropriate journal entriesChatGPT:Thank you for providing those details.Based on the information youve shared,we can make an ini

87、tial assessment:1)The control exception involves a critical control related to preventing users from entering and approving their own journal entries.2)The user in question is the Chief Accounting Officer,whose responsibilities are vital to financial reporting processes.3)Segregation of duties polic

88、ies were in place but were defied by this user,and there were no other mitigating controls.4)The materiality of the impact from the unauthorized journal entries exceeds$4 million,totaling$8 million.5)There was a material weakness identified,as the control deficiency resulted in a significant impact

89、on the financial statements.6)Based on this assessment,it appears that the control exception would be classified as a material weakness,given its significant impact on financial reporting and the absence of effective mitigating controlsContinued from prior slide:29Template PatternTemplate PatternIn

90、this pattern you will tell the LLM to perform a task while giving it a specific outline to follow when generating the output.This prompt saves the user time reworking responses into the format desired.Example:You are an internal auditor documenting a process flow for procurement through payables.You

91、 need to list out the process steps for purchasing a computer through month-end accounting activities using the following template:replacing CAPITALIZED WORDS with the process step.Step#,PROCESS STEP NAME,and summary of that process step.Pattern:I am going to provide a template for your output,“CAPI

92、TALIZED WORDS”is my placeholder for content,replace“CAPITALIZED WORDS”with XYZ.ChatGPT:Certainly,heres an example template for documenting the process flow from purchasing a computer through month-end accounting activities:Step#1:Request for Purchase Summary:The department or individual initiates a

93、request to purchase a computer based on business needs.Step#2:Purchase Requisition Review Summary:The purchasing department reviews the purchase requisition to ensure it complies with budgetary constraints and procurement policies.Step#3:Vendor Selection Summary:The purchasing department selects an appropriate vendor based on factors like price,quality,and delivery timelines.QuestionsQuestions