劉威-LLM在eBay風控RCA中的應用實踐.pdf

1、劉威 eBay高級技術專家eBay 風控部門高級技術專家。曾先后主導了eBay風控實時RCA平臺，風控地址標準化與模糊匹配服務，以及高并發高可用風控名單管理系統的設計與開發工作，負責多個eBay風控決策引擎核心組件的設計，開發與維護，具有豐富的高性能系統架構設計的工程經驗。此外，在知識圖譜與風控決策引擎的集成，大語言模型在風控領域的應用等方向具有深厚的技術積累與豐富的實踐經驗。演講主題：LLM在eBay風控RCA中的應用實踐Empower Risk RCA with LLMLiu,Wei eBay Payments&RiskeBay Risk RCARCA Engine 1.0(RCA+LLM

2、)RCA Engine 2.0(RCA*LLM)Discussion&Future WorkseBay Risk RCAeBay Risk RootCauseAnalysiseBay Risk Decision Platform Business FlowActionsOther9%Traffic4%SignIn5%Holds6%IRSF12%Rules14%Pre Auth38%Risk Decision Platform Alerts8101 per year,40per workday An Example-InternationalRevenue ShareFraud Fraudste

3、rsPhone CarriersSMSPhone number with premium rate SMS from small countrieslease$/monthChallenges on Manual Alert Analysis Initial CheckSite AnalysisRule AnalysisFraud Pattern DetectionUser Sample CheckBusiness InvestigationIncident AlertTake ActionHours for a seasoned analyst Challenges on Manual Al

4、ert Analysis SignInListingOfferCheckout1M+CombinationsMainUSUKDEBlockRemedyFlagApproveActiveNewGuestInactive1133423124621112358291423564RCAChallenges on Manual Alert Analysis Complex interdependencies among different risk elements Numerous data sources need to be individually checked and analyzedHig

5、hly reliant on human knowledge and experience for analysis and decision-makingRCA Engine 1.0Prerequisite-Risk Lineage Graph RuleFlowActionVarSiteModelServiceFeatureDomainUserVendorDW-Primary Nodes-Secondary NodesPrerequisite-Risk Lineage Graph 100K+Nodes2M+EdgesRCA Engine 1.0 Metadata Driven Generic

6、 Pipeline FlowActionSiteRuleUserVariableRCA Pipeline ContextElastic Data FetcherAD Algorithm(e.g.ARIMA)Shift Contribution RankingConditional DecisionRCA Engine 1.0RCA+LLMRCA Engine 1.0 LLM Assisted Detection FlowActionSiteRuleUserVariableRCA Pipeline ContextLog Data FetcherSystem LogLLM DetectionSys

7、tem Error:Downstream service timeoutRCA Engine 1.0 LLM Assisted Decision Making Incident AlertRCA EngineGeneric Pipeline ExecutorData FetcherSherlockElasticRCA MetadataLineage GraphMonitor DataConfigurationManual Verify5 MinsTake Action13 MinsRCA Engine 1.0 LLM Assisted Decision Making Incident Aler

8、tRCA EngineGeneric Pipeline ExecutorData FetcherSherlockElasticRCA MetadataLineage GraphMonitor DataConfiguration13 MinsTake ActionLLM Assisted Decision Making15 sRCA Engine 1.0 LLM Assisted Weekly Report Filter alerts of last weekGroup by categoryIncident AlertsIncident AlertsBuyer RiskSeller RiskA

9、ccount RiskRCA Engine 1.0 Current Adoption 80%61063mAnalysis Tasks in past yearAlerts CoverageAverage Execution TimeRCA Engine 2.0RCA Engine 2.0 Inspiration&Challenge RCA Engine 1.0Generic Pipeline ExecutorData FetcherSherlockElasticRCA MetadataLineage GraphMonitor DataConfigurationRCA Engine 2.0?Da

10、ta FetcherSherlockElasticRCA MetadataLineage GraphMonitor DataConfigurationLLM Driven RCA PlannerSOPsToolboxRCA Engine 2.0 Inspiration&Challenge RCA Engine 2.0?Data FetcherSherlockElasticRCA MetadataLineage GraphMonitor DataConfigurationLLM Driven RCA PlannerSOPsToolboxReasoning CapabilityData Avail

11、abilitySignificant effort is required to integrate with various downstream data sources and to provide tools for LLMs.LLMs often make errors when generating request payloads,which can lead to data server issues and disrupt the processing chainExisting LLM reasoning frameworks(e.g.,ReAct)perform poor

12、ly in complex scenarios with numerous interdependencies among elements.RCA Engine 2.0Project BennuBennu LLM Oriented Data Hub GraphQLRESTPromptControl LayerTiny LLMHelperMonitor&LoggerRouting&FetchingBennu ServiceDiscoveryTestingOnboardingManagingBennu PortalStructured DataRisk VariablesDomain SvcsU

13、nstructured DataWikiGitmailLineage GraphVector DBsPublic DBsShared DBs LabelsLabel DBsClientsCacheAccess ControlRate LimiterBennu LLM Oriented Data Hub TaskLLMData Access ToolsGitWikiDocsLogsMetadataOther ToolsParserEnvironmentReasoningActionResultTaskLLMData Access ToolsGitWikiDocsLogsMetadataOther

14、 ToolsParserEnvironmentReasoningActionResultBennuBennu More than Data Service Faster TTM for LLM AppMinutes to create a new LLM AppKnowledge Sharing&Re-useDiscover&reuse data sources that are already availableBetter Reliability&UsabilityAutomatic error reporting and correction suggestionFully Autono

15、mous LLMPrompt-In&Prompt-Out(PIPO)mechanism to close the loop of LLM thought chainTaskLLMData Access ToolsGitWikiDocsLogsMetadataOther ToolsParserEnvironmentReasoningActionResultBennuRCA Engine 2.0Reasoning on GraphReasoning on Graph Overall Mechanism ReActReasoning on GraphReasoning on Graph for RC

16、A Engine 2.0 Reasoning on Graph for RCA Engine 2.0 RCA Engine 2.0RCA*LLMRCA Engine 2.0 RCA*LLM RCA Engine 2.0Bennu Data Access LayerLLM Driven RCA PlannerSOPsStructured DataRisk VariablesDomain SvcsUnstructured DataWikiGitmailLineage GraphVector DBsPublic DBsShared DBs LabelsLabel DBsToolboxDiscussi

17、on&Future WorksWhile LLMs may occasionally make mistakes,they remain highly valuable toolsRCA Engine 1.5RCA Engine 1.5Bennu Data Access LayerLLM Driven Pipeline GeneratorSOPsStructured DataRisk VariablesDomain SvcsUnstructured DataWikiGitmailLineage GraphVector DBsPublic DBsShared DBs LabelsLabel DBsGeneric Pipeline ExecutorRisk Oops Instructed LLMBase ModelsFine tuned Risk Expert LLMThank you