DatabricksIQ 如何提高工作效率.pdf

《DatabricksIQ 如何提高工作效率.pdf》由會員分享，可在線閱讀，更多相關《DatabricksIQ 如何提高工作效率.pdf（32頁珍藏版）》請在三個皮匠報告上搜索。

1、2024 Databricks Inc.All rights reserved1How DatabricksIQ How DatabricksIQ boosts your boosts your productivityproductivityWeston Hutchins,Sr.Manager,ProductWeston Hutchins,Sr.Manager,ProductRichard Tomlinson,Director of Product MarketingRichard Tomlinson,Director of Product Marketing2024 Databricks

2、Inc.All rights reserved2024 Databricks Inc.All rights reservedIntroductionIntroduction22024 Databricks Inc.All rights reservedAn open,unified foundation for all your dataData LakehouseEasily scale and use data and AIGenerative AIDemocratize data+AI across your entire organizationData Intelligence Pl

3、atform32024 Databricks Inc.All rights reservedAll Raw Data(Logs,Texts,Audio,Video,Images)ETL&Real-time AnalyticsOrchestrationData WarehousingData Science&AIMosaic AIDelta Live TablesWorkflowsDatabricks SQLUnified security,governance,and catalogingUnity CatalogUnified data storage for reliability and

4、 sharingDelta Lake UniFormDatabricks Data Intelligence PlatformDatabricks Data Intelligence PlatformOpen Data LakeOpen Data Lake2024 Databricks Inc.All rights reservedETL&Real-time AnalyticsOrchestrationData WarehousingData Science&AIMosaic AIDelta Live TablesWorkflowsDatabricks SQLUnified security,

5、governance,and catalogingUnity CatalogDatabricks Data Intelligence PlatformDatabricks Data Intelligence PlatformUnified data storage for reliability and sharingDelta Lake UniFormThe AI powered data intelligence engine that understands the semantics of your dataDatabricksIQDatabricksIQOpen Data LakeO

6、pen Data LakeAll Raw Data(Logs,Texts,Audio,Video,Images)2024 Databricks Inc.All rights reservedETL&Real-time AnalyticsOrchestrationData WarehousingData Science&AIMosaic AIDelta Live TablesWorkflowsDatabricks SQLUnified security,governance,and catalogingUnity CatalogDatabricks Data Intelligence Platf

7、ormDatabricks Data Intelligence PlatformUnified data storage for reliability and sharingDelta Lake UniFormThe AI powered data intelligence engine that understands the semantics of your dataDatabricksIQDatabricksIQOpen Data LakeAll Raw Data(Logs,Texts,Audio,Video,Images)Unity CatalogUnity CatalogSecu

8、rely get insights in natural languageDelta Lake UniFormDelta Lake UniFormData layout is automatically optimized based on usage patternsDatabricks SQLDatabricks SQLText-to-SQLText-to-VizWorkflowsWorkflowsJob cost optimized based on past runsDelta Live TablesDelta Live TablesAutomated data qualityMosa

9、ic AIMosaic AICreate,tune,and serve custom LLMs2024 Databricks Inc.All rights reserved7The AI-Powered Data Intelligence EngineDatabricksIQDatabricksIQAI that uniquely understands youryour data and usage patternsBoosts productivity for everyone through natural language interfacesEnables the automatic

10、 optimization of your data and AI workloads2024 Databricks Inc.All rights reservedDatabricksIQDatabricksIQ-Powered ToolsPowered ToolsBoosts productivity for everyone through natural language interfaces8For Data TeamsFor Business TeamsAssistantMetadataSearchAI/BI DashboardsAI/BI Genie2024 Databricks

11、Inc.All rights reserved2024 Databricks Inc.All rights reservedDatabricksIQ DatabricksIQ IngredientsIngredients92024 Databricks Inc.All rights reservedLLMs need great retrieval to workLLMs need great retrieval to workSearch is the foundation foundation of DatabricksIQof DatabricksIQ and is the key to

12、 having relevant,accurate responsesThe assistant without good search2024 Databricks Inc.All rights reservedIn-product context(e.g.notebook content)Databricks documentation(hybrid vector search)Table search(using Databricks APIs)Query table(parsing SQL query and using UC metadata API)DataFrame schema

13、s Previous conversations11Retrieval in the AssistantRetrieval in the AssistantContext is kingContext is king2024 Databricks Inc.All rights reservedYou are the Databricks assistant.The user is diagnosing an error in the Databricks notebook and is looking for a solution.This can either be an explanati

14、on of the problem with the fixed code or steps to execute outside the code to resolve the problem.Here is the cell that was executed:python CELL CONTENT Here are the cells that come before this cell:CELLS BEFORE Here are the cells that come after:CELLS AFTER Error message:ERROR MESSAGE Here are some

15、 documentation pages that could help solve the problem:DOC 1 DOC 2 12Prompt EngineeringPrompt Engineering2024 Databricks Inc.All rights reservedWhat about the models?What about the models?132024 Databricks Inc.All rights reservedVanilla OSS Model FineFine-tuning with Databricks examplestuning with D

16、atabricks examplesNote:In Databricks,you cannot directly delete a column from a table.Instead,you need to create a new table with the desired columns and then drop the original table.Heres an example of how you can achieve this:Step 1:Create a new table with the desired columns CREATE TABLE main.eng

17、.logs_new ASSELECT FROM main.eng.logs;Step 2:Drop the original tableDROP TABLE main.eng.logs;Step 3:Rename the new table to the original table nameALTER TABLE main.eng.logs_new RENAME TO main.eng.logs;.Delete a column in Delta Lake?Delete a column in Delta Lake?142024 Databricks Inc.All rights reser

18、vedVanilla OSS Model FineFine-tuning with Databricks examplestuning with Databricks examplesNote:In Databricks,you cannot directly delete a column from a table.Instead,you need to create a new table with the desired columns and then drop the original table.Heres an example of how you can achieve thi

19、s:Step 1:Create a new table with the desired columns CREATE TABLE main.eng.logs_new ASSELECT FROM main.eng.logs;Step 2:Drop the original tableDROP TABLE main.eng.logs;Step 3:Rename the new table to the original table nameALTER TABLE main.eng.logs_new RENAME TO main.eng.logs;.Delete a column in Delta

20、 Lake?Delete a column in Delta Lake?15x2024 Databricks Inc.All rights reservedVanilla OSS Model FineFine-tuning with Databricks examplestuning with Databricks examplesNote:In Databricks,you cannot directly delete a column from a table.Instead,you need to create a new table with the desired columns a

21、nd then drop the original table.Heres an example of how you can achieve this:Step 1:Create a new table with the desired columns CREATE TABLE main.eng.logs_new ASSELECT FROM main.eng.logs;Step 2:Drop the original tableDROP TABLE main.eng.logs;Step 3:Rename the new table to the original table nameALTE

22、R TABLE main.eng.logs_new RENAME TO main.eng.logs;.Delete a column in Delta Lake?Delete a column in Delta Lake?16Fine-tuned on Databricks documentationSQL:ALTER TABLE main.eng.logsDROP COLUMN _partition_backfill_version;x2024 Databricks Inc.All rights reservedHard to find realistic eval setsSupervis

23、ed LLM-as-judgeIn-product side-by-side resultsHuman-in-the-loop labeling and curation17EvaluationEvaluationMeasuring qualityMeasuring quality2024 Databricks Inc.All rights reservedIs it better to have a faster answer with occasionally lower accuracy or a slower,more accurate answer?It depends!Diagno

24、se error=Accuracy is keyAI Autocomplete=Latency is key18LatencyLatencyQuality vs.responsiveness Quality vs.responsiveness 2024 Databricks Inc.All rights reservedRetrievalPrompt EngineeringFine-tuned ModelsEvaluationLatencyAnd many more:UX,personalization,knowledge graphs,NL parsing,DatabricksIQ Ingr

25、edientsDatabricksIQ Ingredients19HighHigh-quality AI experiences quality AI experiences 2024 Databricks Inc.All rights reserved2024 Databricks Inc.All rights reservedBoost productivityBoost productivityfor data teamsfor data teams202024 Databricks Inc.All rights reservedDatabricksIQDatabricksIQ-Powe

26、red ToolsPowered ToolsBoosts productivity for everyone through natural language interfaces21For Data TeamsFor Business TeamsAssistantGovernanceSearchAI/BI DashboardsAI/BI Genie2024 Databricks Inc.All rights reserved22DemoDemo2024 Databricks Inc.All rights reservedAuto-fix errorsMore inline experienc

27、es(e.g.,rename,refactors,quick refinement)Performance optimizationDiagnose job errorsCatalog Explorer help23Coming enhancementsComing enhancements2024 Databricks Inc.All rights reserved2024 Databricks Inc.All rights reservedBoost productivityBoost productivityfor business teamsfor business teams2420

28、24 Databricks Inc.All rights reservedDatabricks Data Intelligence PlatformDatabricks Data Intelligence PlatformThe AI powered data intelligence engine that understands Data Intelligence EngineData Intelligence EngineOpen Data LakeOpen Data LakeAll Raw Data(Logs,Texts,Audio,Video,ImaWorkfloWorkfloJob

29、 cost op based on pa Delta Live TablesDelta Live TablesAutomated data qualityUnity CatalogUnity CatalogSecurely get insights in natural langDelta Lake UniFormDelta Lake UniFormData layout is automatically optimized based on Mosaic AIMosaic AICreate,tune,and serve custom LLMsOperationsFinanceMarketin

30、gCustomer ServiceHow can we extend data and AI to everyone in the organization?“Databricks AI/BI”Data and AI for all with natural language2024 Databricks Inc.All rights reservedDatabricks AI/BIDatabricks AI/BIIntelligent analytics for real-world dataGenerally AvailablePublic PreviewDashboardsGenieDa

31、shboardsGenie2024 Databricks Inc.All rights reservedGENIEGENIEQuery AgentDatabricksIQDatabricksIQQuery AgentAI AgentsSemanticUnderstandingCompound AI System that Compound AI System that continuously learns your continuously learns your unique data and semanticsunique data and semanticsEnsemble of sp

32、ecialized AI agents working in concert Seeks clarifications when neededLeverage tools and contextUnity Catalog metadataQuery history(all workloads)Related assets(Notebooks,Dashboards,Queries)Remembers and learns2024 Databricks Inc.All rights reserved28DemoDemo2024 Databricks Inc.All rights reservedW

33、ant to learn Want to learn more about more about DatabricksIQ?DatabricksIQ?292024 Databricks Inc.All rights reserved30Visit Visit read the blogs(links on web page)And read the blogs(links on web page)2024 Databricks Inc.All rights reservedEnabling DatabricksIQEnabling DatabricksIQ-Powered ToolsPowered Tools31Ensure your Admins Ensure your Admins have enabled DBIQ have enabled DBIQ featuresfeatures2024 Databricks Inc.All rights reserved2024 Databricks Inc.All rights reservedThanks!Thanks!32