構建數據平臺的語義層.pptx

《構建數據平臺的語義層.pptx》由會員分享，可在線閱讀，更多相關《構建數據平臺的語義層.pptx（51頁珍藏版）》請在三個皮匠報告上搜索。

1、BUILDING A SEMANTIC LAYER OF YOUR DATA PLATFORMDATA SUMMIT WORKSHOPAPRIL 2024Joe Hilger,COO and Principal(jhilgerenterprise-)Sara Nash,Principal Consultant(snashenterprise-)What You Will Learn Today1.The business case for semantic layers 2.Definitions of the semantic layer and its components3.Real w

2、orld applications for semantic layers4.How to prototype semantic layer use cases and modelsEKCONSULTINGIntroductionsJoe HilgerCOO and Principal ConsultantEnterprise KnowledgeSara NashPrincipal ConsultantEnterprise KnowledgeEKCONSULTINGWhy Semantic Layers?ENTERPRISE KNOWLEDGEModern Data Platforms Pro

3、vide Answers and Context ENTERPRISE KNOWLEDGEModern Data Platforms Enable Discovery ENTERPRISE KNOWLEDGEModern Data Platforms Increase Access and Trust in Data ENTERPRISE KNOWLEDGEModern Data Platforms Break Down Silos ENTERPRISE KNOWLEDGEHow are these experiences enabled?Content(knowledge,data,and

4、information)is managed and accessible Data is connected across repositories,databases,and applications Context and meaning is embedded with source data,making common understanding of data machine-readableENTERPRISE KNOWLEDGEWhat is a Semantic Layer?“A semantic layer is a standardized framework that

5、organizes and abstracts organizational knowledge and data(structured,unstructured,semi-structured)and serves as a connector for all organizational knowledge assets.”The Semantic Layer Your Datas“Rosetta Stone”A semantic layer encodes information based on context and business logic via a broader unde

6、rstanding of what those data values mean to an organization.The components of the semantic layer(Knowledge Graph,Taxonomy,Metadata Service,etc.)work together to translate data across systems,enabling comprehensive analytics,reporting,and search applications.ENTERPRISE KNOWLEDGEINCONSISTENT METADATAN

7、ON-INTUITIVE USER INTERACTIONSINEFFICIENT DATA ANALYSIS PROCESSESWhat Problems Does a Semantic Layer Solve?POOR DATA QUALITY&GOVERNANCE VENDOR LOCKAI HALLUCINATIONSENTERPRISE KNOWLEDGEStandardizing MetadataUnified language used to talk about information across the organizationConnecting InformationS

8、tructured and unstructured data and content are both human and machine-readable.Utilizing AI&LLMsAI-supported applications like LLMs perform with higher fidelity and avoid hallucinationsReporting and AnalyzingSearch results display information that exists in multiple locations and formats producing

9、a one-stop data hubWhat Does a Semantic Layer Enable?Foundations of Semantic Layers:ComponentsDescribeStandardizeCatalogMetadataProduct Name:iPhone 7Product Type:ElectronicsProduct Dataset:prod_data Date Created:02-20-2024What data do we have on our products?How can we quickly identify and retrieve

10、product images for our marketing campaigns?Which datasets are capturing profitability from product Sales?DefineAlignIndexBusiness GlossaryTermTypeDefProfitability INTThe degree to which a product sale yields financial gainHow do we define profitability for a product area?What is the meaning of custo

11、mer journey in our company,and how is it mapped?What is the agreed-upon definition of key performance indicator(KPI)in our organization?CategorizeOrganize Architect Taxonomy|Information Architecture ElectronicsTVs Cell Phones Which products fall under this Product category?How can we better organize

12、 our product content to enhance customer service efficiency?How can we improve the organization of our online product website for a smoother customer browsing experience?RelateMap ContextualizeOntologyProductRegionCustomerSaleHasCustomerHasSaleHasLocationLivesInHow can we ensure a unified understand

13、ing of product data across departments?What are the sequential steps in our supply chain process,and how are they interrelated?How do Online Sales relate to Regions?ConnectAnalyze InferKnowledge GraphiPhone7 NorthEastJoebot$What is the most purchased product across customers in a given region?What i

14、s our revenue from this product in Store A in 2023?How does a disruption in one part of our supply chain affect other components?Foundations of Semantic Layers:Enabling TechnologyENTERPRISE KNOWLEDGESemantic LayerAPIsETLApplicationSearchResearch&AnalyticsRecommendations&ChatbotsAdmin&GovernanceKnowl

15、edge GraphMetadata ServiceBusiness GlossaryTaxonomyRDF Ontology/LPG SchemaContent Management SystemData Lake/Data WarehouseSubscriptionsExternal SourcesData SourcesPresentation LayerAPIsHigh Level ArchitectureENTERPRISE KNOWLEDGESemantic LayerAPIsETLApplicationSearchResearch&AnalyticsRecommendations

16、&ChatbotsAdmin&GovernanceKnowledge GraphMetadata ServiceBusiness GlossaryTaxonomyRDF Ontology/LPG SchemaContent Management SystemData Lake/Data WarehouseSubscriptionsExternal SourcesData SourcesPresentation LayerAPIsSemantic Layers and AI Fact-based natural language query Relationship-based recommen

17、dationsAdvanced analytics Extract entities and relationshipsSummarize unstructured data Recognize patterns and curate factsENTERPRISE KNOWLEDGEWhat Is NOT a Semantic Layer?Not Just for DataNot A Single ProductNot Just a GraphNot All of Your ContentNot an Automated/AI-Generated SolutionWhy Our Client

18、s are Investing in Semantic Layer ENTERPRISE KNOWLEDGETop Enterprise Use CasesENTERPRISE KNOWLEDGEReporting and InsightsUse Case 1ENTERPRISE KNOWLEDGEThe ChallengeThe Solution The solution provided a 360 view of data used for analysis across all systems The knowledge graph enabled automated creation

19、 of reports for regulatory filings and analysis of lots of different process details in depth.The ResultsAn organizations team of scientists needed to quickly find and get insights about drug development processes.However,their insights were limited to what the scientists could manually aggregate fr

20、om siloed legacy systems with different naming conventions.Develop a comprehensive ontology to model the drug development process and standardized nomenclature.Build a knowledge graph that aggregated and normalized disparate data from legacy systems.Use Case 1:Global Biotechnology Company ENTERPRISE

21、 KNOWLEDGEUse Case 1:Global Biotechnology Company ENTERPRISE KNOWLEDGEUse Case 1:Global Biotechnology Company What is the dataset we used for this Experiment?How do we define a Product Run?What Materials did we use to create this Product?What was the Average Cell Viability for this Product?What are

22、the Measurements used in this Experiment?What are our Products?Which Business Unit led this Project?ENTERPRISE KNOWLEDGEUse Case 2Enterprise Data ArchitectureENTERPRISE KNOWLEDGEThe ChallengeThe Solution The semantic model allowed consistent categorization of risk data across 15+systems.Deduplicated

23、 risk descriptions by 40%using AI-based approaches.The centralized and standardized structure for data makes it easier to connect and handle data on a large scale.The ResultsA multinational financial services firm needed strategy support in implementing a data modernization program to solve their or

24、ganizations problems with risk identification and the user experience around reporting.Build central taxonomy network to provide consistent categorization of risk data across systems.Develop semantic model for data connectivity to drive knowledge panels,recommendation engines,and semantic search.Lev

25、erage LLMs to standardize risk category descriptions.Use Case 2:Financial Services FirmAssessmentUse Case 2:Financial Services FirmRisks Process OrganizationBusiness Units Legal Entities RegionsProducts/ServicesControlsIncidentsCompliance Rules PoliciesComplaintsRegulatory MattersMonitoring&Testing

26、Priority Risk ProgramsIdentify MitigateGovernance,Procedures,and Training Data,Metrics,and ReportingKey Takeaways Enterprise Data-warehouse to drive data aggregation for reporting and analytics Shared Services for authentication,entitlements,and logging Semantic Services and Storage to support graph

27、 use cases such as semantic search,personalization,data quality and connectivity Data Provider Model for semantic data exchange between business applications and semantic layerEnterprise ArchitectureUse Case 2:Financial Services FirmENTERPRISE KNOWLEDGEUse Case 2:Financial Services FirmWhat is the R

28、ule relevant to this Risk?What is a Business Unit vs.a Product?What Controls are mitigating this Risk?What are the Risk Ratings for this Assessment?What are our Products?How should we describe our Controls?ENTERPRISE KNOWLEDGEUse Case 3Data Modernization&User JourneysENTERPRISE KNOWLEDGEThe Challeng

29、eThe Solution Enabled creation of a semantic layer to support organization-wide data discoverability Provided a deep understanding of the targeted improvements to data accessibility that would enhance user satisfaction,engagement,and efficiency Identified short,medium,and long term goals for reachin

30、g the defined target stateThe ResultsA global retail chain needed support in confronting challenges with finding,connecting to,and understanding existing data related to tracking the health of their stores.Create business glossary and taxonomy to tag business assets for increased findability and und

31、erstanding Create journey maps describing current state and target state persona experiences.Design detailed roadmap outlining key steps for improving organizational semantic maturityUse Case 3:Global Retail ChainEnd StateIndependent understanding the business meaningMinutes to see results Data demo

32、cratization and ownership Current StateHeavy reliance on the IT data analytics teamWeeks of discussion to get results High volume of data USER STORY:As an Executivewant to understand the business meaning of data so that I can make a quick and informed decision about what business strategy my team sh

33、ould use.Step 1Executive reads a data analytics report.Step 2Executive doesnt understand what the data is trying to represent.Step 3Executive approaches data analytics team asking for explanation.Step 4Executive waits for data analytics team to consult about the meaning of data included in the repor

34、t.Step 5Executive receives an explanation from the data analytics team about data meaning.Step 6Executive can make a strategic decision based on the meaning of the data in the report.Step 1Executive reads a data analytics report.Step 2Executive looks up the report in a UI connected to the Semantic L

35、ayer.Step 3Executive finds business definitions of the data in the report.Step 4Executive can make a decision based on the meaning of the data in the report.Use Case 3:Global Retail ChainUser Journey MapEnd StateMinutes to find and understand tables Self-directed discovery to vet data Current StateW

36、eeks to find and understand tablesMeetings to determine if the tables can be usedUSER STORY:As a Data Analytics Team MemberI want to find and understand relevant data tablesso that I can easily reuse existing data for a project.Step 1DA Team Member is assigned a project that requires data analytics.

37、Step 2DA Team Member spends hours to days looking through familiar data repositories for data tables that could be useful.Step 3DA Team Member is unable to find what they need,and asks a SME for instructions on where to look.Step 4DA Team Member spends weeks looking through recommended data reposito

38、ries,and finds tables that seem relevant.Step 5DA Team Member does not understand the tables columns,and asks a SME for definitions and calculation rules.Step 6DA Team Member is able to determine if the existing data table is suitable for their project.Step 1DA Team Member is assigned a project that

39、 requires data analytics.Step 2DA Team Member searches for data tables in a UI connected to the Semantic Layer.Step 3DA Team Member finds data tables with metadata and column definitions.Step 4DA Team Member is able to determine if the existing data table is suitable for their project.Use Case 3:Glo

40、bal Retail ChainUser Journey MapENTERPRISE KNOWLEDGEUse Case 3:Global Retail ChainWhere do I find the latest Performance Reports?What is the definition of a Data Product?Who are the domain Experts?What caused a Store Outage?Which Metrics does a given Store need to track?What Topic do I use to tag my

41、 Data?ENTERPRISE KNOWLEDGEUse Case 4Data Consistency and UsabilityENTERPRISE KNOWLEDGEThe ChallengeThe Solution The contribution model enabled 10+departments to contribute and lead semantic model development and governance.Drove the implementation of data standards through the publication of the ent

42、erprise ontology.Increased data awareness,consistent understanding,and alignment for users across departments and technologies.The ResultsA large financial corporation had a lack of alignment around the meaning,format,and intent of data elements across organizational divisions,reducing the ability o

43、f data producers and consumers to find,use,and and trust data.Develop an enterprise ontology to standardize data from multiple systems and migrate from an existing physical data model.Implement a federated ontology governance and contribution model.Leverage standardized ontology concepts throughout

44、the data lifecycle.Use Case 4:Financial Services CompanystateCodeCountrySubdivisionCodeSubdivision_Codestate_postal_codeUSPS_CodeSTATEUSALUSAKUSAZcountryCodeIsoCountryCodealpha_code_countrygeo.country_code2iso_3166_country_code_2CTRYEKCONSULTINGUse Case 4:Financial Services CompanyUse Case 4:Financi

45、al Services CompanyENTERPRISE KNOWLEDGEWhere can I access Compliant Data Models?How do we define Credit Risk across departments?What Legacy Systems were integrated?How is risk data Classified?How is Data structured for Regulatory Compliance?Use Case 4:Financial Services CompanyUse Case 4:Financial S

46、ervices CompanyLets Roll Up Our Sleeves ENTERPRISE KNOWLEDGEActivity 1:Use Case DefinitionEstablishing Your Prototype Use Case:SampleSemantic Layer Use CaseUser StoryAs an online learner,I want to see course recommendations when I get an assessment question incorrect,so that I can upskill in that ar

47、ea of weakness.SpecificationsSource Data Assets Course Library Question and Assessment Library Healthcare Topic Taxonomy Key Knowledge Concepts Course Question Topic Healthcare SettingEKCONSULTINGENTERPRISE KNOWLEDGEActivity Instructions1.As a group,complete 3-5 use case worksheets(15 minutes).You c

48、an select a use case from your experience or make up examples.1.Each group will present their a few of the use cases we will select favorites to model.2.In our next activity,you will be modeling your selected use case.EKCONSULTINGENTERPRISE KNOWLEDGEActivity 2:ModelingHow to Model Your Knowledge Gra

49、phEKCONSULTING Entity A unique type of thing that you want to define and relate in your model.Attributes or properties.Aspects,features,characteristics,descriptors,or parameters that describe and differentiate instances of an entity.Relationships The types of connections that can be defined between

50、entities.PersonWorks ForCompanyAddressFounding DateAnnual RevenueSellsActivity Instructions Define EntitiesWrite the Entity names on hexagons.Define RelationshipsUsing string and tape,define Relationships by connecting two Entities.Define AttributesWrite the Attribute names on sticky notes and stick

51、 to the relevant Entity.With your group,build out the semantic layer model for your use case using the materials provided.Definition of Done:The data concepts required to support your use case are incorporated.CustomerCustomerProductCustomerNameAgeBuysCustomerNameAgeProductSizeColorBuysProduct FamilySizeColorpartOfEKCONSULTINGJHILGERENTERPRISE-KNOWLEDGE.COMJoe HilgerWWW.LINKEDIN.COM/IN/JOSEPH-HILGER/SNASHENTERPRISE-KNOWLEDGE.COMSara NashWWW.LINKEDIN.COM/IN/SARA-G-NASHContact UsEKCONSULTING