AI 和湖倉一體：殼牌邁向有效數據治理的旅程.pdf

1、2024 Databricks Inc.All rights reserved1AI&THE LAKEHOUSE:SHELLS JOURNEY TOWARDS EFFECTIVE DATA GOVERNANCEJohn OBrienJohn OBrienJune 2024June 2024AI&THE LAKEHOUSESHELLS JOURNEY TOWARDS EFFECTIVE DATA GOVERNANCEJohn OBrienProduct Manager-Data,Shell Energy Australia T&SShell Energy Trading is a global

2、leader in trading gas,power,and environmental products.We bring decades of marketing and trading experience to offer more and cleaner energy solutions to our customers.We work in partnership with Shell businesses across oil regions to offer energy solutions that help our customers on their decarboni

3、sation journey.Technology underpins all that we do to improve our performance and competitiveness.SHELL ENERGY TRADING(SET)Supported by Shells global supply portfolio and expansive trading networkhttps:/.auWhy is good data governance now even more important?Generative AI requires quality data to pro

4、vide context5AgendaInitial Difficulties in Data Strategy and Governance01This is HardUnity Catalog and Business Owned Data Products02Our SolutionOur implementation and the use of Generative AI for governance03Data MeshLessons on Analytics,PowerBI,ML and Generative AI for governance04Top LessonsHow w

5、ill we scale this and next steps05TakeawaysThis is hardIve created 3 data strategies and 2 refreshes in 12 years.People are busy,often not the highest priority for themOften seen simply as a nice to have rather than essentialHaving tangible goals that all stakeholders can value is keyExample Data St

6、rategy Phased Approach for SuccessData CentralisationEnable Platform and first dataset for high value user casesConnect in priority order all the existing dataImprove ingestions to streamingData DemocratisationEnable bronze access for everyoneEnable workflow to give access to Silver and Gold to ever

7、yoneExtend to Volumes,Model and moreData MonitoringJobs Monitoring(proactive)Data Quality MonitoringAnomaly DetectionData ProductsStart small(e.g.5)Expand(e.g.20)Organically grow(e.g.40)Data Science1 ML ModelProve the pattern works(e.g.3)Organically grow(e.g.12)Data SharingCatalogs in RegionDelta Sh

8、aringLakehouse FederationData&Data Products GovernanceUse all the features of Databricks Unity CatalogCentral Tooling IntegrationAI AgentsOur solutionFocus on immediate value with longer term strategyGood tools that work togetherIts okay to not have everything readyBusiness ownership Shell Energy Au

9、stralia T&S on Databricks LakehouseStart small then grow/scale up6 users creating 6 data productsGrew to 120 users and 20 data products in 12mth.Pattern for 800 users in tradingProvide value from the beginningSingle sign on access to dataHigh performance computeSource control/CollaborationHigh quali

10、ty toolingDatabricks Unity CatalogGithubPowerBI and Plotly Dash EnterpriseAgile approachProvision template,schema,repo,cluster,Security GroupBuild valueRegular improvementsData Product Tooling FrameworkRead only access to ingested data from the sourcesRaw business developed data productAgile approac

11、h to data modellingOwnership with the business,IT as the enablerFull suite of Training and SupportSchema with write accessGithub RepositoryWorkspace for Collaboration and AlertsCompute Cluster(optional ML)Listing in CatalogData Product Tooling Framework DemoData meshData as a productSelf-service inf

12、rastructure as a platformFederated GovernanceHow to effectively govern a data mesh?OwnershipData catalogData linageData securityTop lessons1.Need to be able to deliver the foundations quickly2.Landscape is constantly changing,dont wait for perfect3.Focus on value,prioritise these features4.Give your people superpowersGenerative AI for metadata Databricks Code DemoAI generated image DALLE-3TakeawaysHow does this scale across a diverse organisation?Where do you begin?Why Generative AI is the trigger to deliver this.Q&A