探索 MLOPS 和 LLMOPS：架構和最佳實踐（重復）.pdf

《探索 MLOPS 和 LLMOPS：架構和最佳實踐（重復）.pdf》由會員分享，可在線閱讀，更多相關《探索 MLOPS 和 LLMOPS：架構和最佳實踐（重復）.pdf（45頁珍藏版）》請在三個皮匠報告上搜索。

1、2024 Databricks Inc.All rights reservedExploring MLOps Exploring MLOps and LLMOps:and LLMOps:Architectures Architectures and Best and Best PracticesPractices1Arpit Jasapara,Yinxi Zhang,Joseph BradleyArpit Jasapara,Yinxi Zhang,Joseph BradleyDatabricksDatabricks2024 Databricks Inc.All rights reserved2

2、About usAbout usArpit JasaparaArpit JasaparaEngineeringYinxi ZhangYinxi ZhangData ScienceJoseph BradleyJoseph BradleyProduct Specialist2024 Databricks Inc.All rights reservedMLOpsTalk(20 minutes)MLOps Stacks Demo(15 minutes)Q&A(10 minutes)LLMOpsTalk&MosaicAI Agent Framework Demo(35 minutes)Q&A(10 mi

3、nutes)3AgendaAgendaMLOps+LLMOpsMLOps+LLMOps2024 Databricks Inc.All rights reservedWhat is MLOps?Why should I care?Best practicesMLOps on DatabricksUnity CatalogModel ServingMonitoringDatabricks MLOps Stacks MLOps Stacks Demo4MLOpsMLOps2024 Databricks Inc.All rights reservedMLOps is the set of proces

4、ses and automationprocesses and automationfor managing data,code,and modelsmanaging data,code,and modelsto improve performance,stability,and efficiencyperformance,stability,and efficiency of ML systems5What is MLOps?What is MLOps?MLOps=DataOps+DevOps+ModelOpsMLOps=DataOps+DevOps+ModelOpsBig Book of

5、MLOps2024 Databricks Inc.All rights reserved6ModelOpsDataOpsDevOpsUnity CatalogWorkflowsModel ServingLakehouse Monitoring2024 Databricks Inc.All rights reservedMLOps helps you reduce riskTechnical risk-poorly performing models,fragile infrastructureCompliance risk-violating regulatory or corporate p

6、oliciesMLOps improves long-term efficiency through automationStreamline delivery of models to productionCatch errors before they hit productionAvoid slow,manual processes7Why should I care about MLOps?Why should I care about MLOps?2024 Databricks Inc.All rights reserved8Best Practices:Roles and Proc

7、essBest Practices:Roles and Process Builds data pipelines Translates business problem;trains,tunes model Deploys ML model to production Responsible for data governance and compliance Responsible for business value of the ML solutionBusiness StakeholderData EngineerData Scientist(DS)ML Engineer(MLE)D

8、ata Governance Officer2024 Databricks Inc.All rights reserved9Best Practices:Roles and ProcessBest Practices:Roles and Process2024 Databricks Inc.All rights reservedBest Practices:Environment IsolationBest Practices:Environment Isolation10AI Assets:DevelopedTestedDeployedCodeDataModelsAssets need to

9、 be:devstagingprodFeaturesLabelsMLflow Model(sklearn,)TrainingInferenceTrust,quality,and testing:LowHighOpenness of access:OpenLocked-down2024 Databricks Inc.All rights reservedBest Practices:Deployment PatternsBest Practices:Deployment Patterns112024 Databricks Inc.All rights reservedBest Practices

10、:Deploy CodeBest Practices:Deploy Code12DevelopmentDevelopmentStagingStagingProductionProductionDevelop training codeTest model training code on subset of dataTrain model on production dataDevelop ancillary codeTest ancillary codeTest model Promote code Promote code Deploy model and ancillary codeAu

11、tomationAutomationSupports automated retraining in locked-down environments.Data access Data access controlcontrolOnly production environment needs read access to production training data.Reproducible Reproducible modelsmodelsEngineering control over training environment,which helps to simplify repr

12、oducibility.Support for Support for large projectslarge projectsThis pattern forces modular code and iterative testing,helping coordination and development.ProcessBenefits2024 Databricks Inc.All rights reservedMLOps on Databricks:Unity CatalogMLOps on Databricks:Unity Catalog13Single governance solu

13、tion for data and AI assets:Centralized access controlAuditingLineage(tables,features,models,workflows,etc.)DiscoverySharing assets between workspaces2024 Databricks Inc.All rights reservedMLOps on Databricks:Model ServingMLOps on Databricks:Model Serving14Automatic feature/vector lookups,monitoring

14、,and unified governanceDatabricks IntegratedDeploy any model type on CPU or GPU with scalable,automated container build andvery low latency(p50 25k)Simplified,Serverless DeploymentSupports online evaluation strategies such as A/B testing through the ability to serve multiple models to a serving endp

15、ointProduction ReadyDeploy models as a real-time API to integrate model predictions with applications or websites.Log each request&response for monitoring,retraining,debugging,and moreInference Tables2024 Databricks Inc.All rights reservedMLOps on Databricks:MonitoringMLOps on Databricks:Monitoring1

16、5Data-centric monitoring solution to ensure that both data and AI assets are of high quality,accurate,and reliable.Incrementally processes data in UC tablesCalculates profile metrics and driftmetricsSupports custom metrics as SQL expressionsAuto-generates DBSQL dashboardto visualize metrics over tim

17、eFor MLOps,use in conjunction with inference tables to monitor models2024 Databricks Inc.All rights reserved162024 Databricks Inc.All rights reservedDatabricks MLOps StacksDatabricks MLOps Stacks17A customizable framework for managing the ML lifecycle on Databricks,following all MLOps best practices

18、 Create and manage production MLOps infrastructure on Databricks Integrates with common CI/CD providers like GitHub and Azure DevOps Manage AI assets(experiments,features,models,monitoring,etc.)and workflows as IaC with Databricks Asset BundlesMLOps Stacks allow you to focus on ML,not infrastructure

19、 DS get started with project development option in Stacks MLEs easily set up CI/CD and customize the architecture as needed via the CI/CD option in Stacks DS then safely deploy project to production through secure CI/CD pipelines and workflows setup by the MLEs2024 Databricks Inc.All rights reserved

20、Databricks Databricks MLOps Stacks MLOps Stacks DemoDemo182024 Databricks Inc.All rights reserved19MLOps Q&AMLOps Q&A2024 Databricks Inc.All rights reserved20What changes with Gen AI and LLMs?Key components for Gen AIArchitecture and AI securityDemoLLMOpsLLMOps2024 Databricks Inc.All rights reserved

21、Reminder:Shipping ML from Dev to Prod21ML Assets:DevelopedTestedDeployedCodeDataModelsAssets need to be:Execution EnvironmentdevstagingprodFeaturesLabelsMLflow Model(sklearn,)TrainingInference2024 Databricks Inc.All rights reservedShipping Gen AI from Dev to Prod22Gen AI Assets:DevelopedTestedDeploy

22、edCodeDataModelsAssets need to be:Execution EnvironmentdevstagingprodFeaturesVector DBMLflow Model(LangChain,)ChainInference2024 Databricks Inc.All rights reservedMLOps MLOps-What changes with Gen AI?What changes with Gen AI?23Properties of Gen AI modelsImplications for MLOpsModels come in many form

23、s:General vs.domain/task-specific modelsProprietary vs.OSSModel-as-a-service APIs vs.self-managedExisting vs.custom fine-tuned vs.custom pretrained modelsDevelopment processLegal concernsAPI governancePackaging artifactsServing infrastructureCustom modelsEvaluationModels range widely in size:Top gen

24、eral models have 100 billions-trillion parametersTop domain/task-specific models may have billions of parametersModels take natural language prompts(or other unstructured data)as input.Models can be given prompts with examples and/or context.Models are hard to evaluate via traditional ML metrics sin

25、ce there is often no single“right”answer.2024 Databricks Inc.All rights reserved24Selecting modelsLeveraging your own dataEvaluating Gen AI systemsDeploying and monitoring systemsLLMOps:Key components for Gen AILLMOps:Key components for Gen AI2024 Databricks Inc.All rights reservedPlan to use a vari

26、ety of modelsWhy?Cost/performance trade-offsTask/domain-specific modelsModel improvementsHow?Unified APIs and governanceToolchain supporting arbitrary models and providersSelecting modelsSelecting models25Key adviceKey advicePlan to build custom modelsWhy?Cost/performance trade-offsTask/domain-speci

27、fic modelsBuilding IP and competitive edgesHow?Collect data and feedback nowChoose models carefully2024 Databricks Inc.All rights reservedUnified API for all types of models.Integration with services including Feature Serving,Vector Search,and Lakehouse Monitoring.Governance via Unity Catalog for mo

28、del objects and AI Gateway for APIs.Selecting modelsSelecting models26Databricks Model ServingCustom ModelsFoundation ModelsExternal ModelsGovern external models and APIs.Curated Foundation Models provided behind simple APIs.Pay-per-token and provisioned throughput,including for fine-tuned versions.

29、Deploy any model as a REST API with Serverless compute,managed via MLflow.CPU and GPU.For model guidance,see for example:Best-in-class open source generative AI models for free commercial use2024 Databricks Inc.All rights reservedLeveraging your own dataLeveraging your own data27More control and cus

30、tomization,but more compute and complexityPrompt EngineeringCraft prompts to guide GenAI behaviorRAG,Agents,and ToolsCombine a GenAI model with custom,enterprise data and toolingFine-tuningAdapt a pre-trained GenAI model to specific domains or tasksPre-trainingTrain a GenAI model from scratchFew-sho

31、t examplesEvaluation dataVector databaseFeature servingFunction servingSQL databaseDomain-specific data(millions of tokens)Task-specific examples(1000s)General and domain-specific data(billions of tokens)2024 Databricks Inc.All rights reservedUnify data governanceWhy?Data will grow:Raw data,context

32、for RAG,inference logs,evaluation metrics,feedback,Data will be reused across use casesHow?Unified management of all types:raw files,tables,embeddings,feature serving,vector indexes,logs,metrics,Unified governance of data+AI assetsLeveraging your own dataLeveraging your own data28Key adviceKey advic

33、ePlan towards customizationWhy?Your data is your competitive edge.How?Work on platforms supporting fine-tuning and pretrainingStart simple,create baselines,iterate.Add customization based on:Volume and quality of dataCompute&latency requirementsYour domain or application2024 Databricks Inc.All right

34、s reservedBuild user feedback into your appWhy?Users can be the best judgesBuild proprietary datasets for future fine-tuning and pretrainingHow?Consider implicit and explicit feedbackManage feedback like any other data:same governance,same ETL,etc.Evaluating Gen AI systemsEvaluating Gen AI systemsAu

35、gment existing eval toolingWhy?Much tooling is reusable:MLflow,data pipelines,etc.New metrics can be added to existing systemsHow?Adopt metrics from classic areas:toxicity(NLP),precision/recall(IR),Use new tools like LLM-as-a-judgeEvaluate both the components+system as a whole29Key adviceKey advice2

36、024 Databricks Inc.All rights reservedInteractive evaluation in UICompare multiple models and prompts visuallyIteratively test new queries during developmentBatch evaluation in codeLLM-as-a-judgeHuman evaluation using ground truth dataNew metrics for Gen AI,NLP,and retrievalEvaluating Gen AI systems

37、 withEvaluating Gen AI systems with302024 Databricks Inc.All rights reservedUse optimized inferenceWhy?User experience and TCOHow?Real-time:Model ServingFoundation Model APIs for pre-optimized architecturesCustom models for DIYBatch and streamingai_query to call Model ServingGPU clusters with vLLM,e

38、tc.Use flexible tooling for packagingWhy?You will swap AI libraries over time:LangChain,LlamaIndex,Python,Uniform APIs lower the cost of switching libraries for a use caseHow?MLflow supports built-in flavors,PyFuncs,and custom flavors.All are managed behind uniform APIs.Deploying and monitoring syst

39、emsDeploying and monitoring systems31Key adviceKey advice2024 Databricks Inc.All rights reservedYour monitoring and core data/AI systems should be unified.Why?Governance,lineage,and security are more important than ever with Gen AI.Inference logs,feedback,and metrics may be inputs for other AI syste

40、ms.How?Unify governance,lineage,and access controls across data(inputs and outputs)and assets(data and AI)in your platform.Share data formats(such as Delta)efficiently usable by all systems.Share data pipeline tooling.Deploying and monitoring systemsDeploying and monitoring systems32Key adviceKey ad

41、vice2024 Databricks Inc.All rights reserved33Reference architecturesPrompt engineeringRAG and agentsFine-tuningPretrainingDatabricks AI Security FrameworkLLMOps:Architecture and AI securityLLMOps:Architecture and AI security2024 Databricks Inc.All rights reserved34Architecture:prompt engineeringArch

42、itecture:prompt engineeringMonitoringLakehouse MonitoringInference TablesLog query,response,metrics5AI AppUsersResponse41QueryWeb AppModel HubModels in Unity CatalogHuggingFace HubChoose and serve LLMModels in MarketplaceCreate PromptsSend promptsModel ServingTemplatesInstructionsExamplesPromptsCust

43、om Models(CPU/GPU)Foundation ModelsExternal Models232024 Databricks Inc.All rights reserved35Reusable infrastructureReusable infrastructureModel HubModel ServingModels in Unity CatalogHuggingFace HubMonitoringLakehouse MonitoringInference TablesAI AppWeb AppCustom Models(CPU/GPU)Foundation ModelsExt

44、ernal ModelsModels in MarketplaceYour initial GenAI use case will help you to assemble key pieces of your eventual GenAI+data platform.2024 Databricks Inc.All rights reserved36Architecture:RAG and agentsArchitecture:RAG and agentsConstruct Prompts3TemplatesPromptsRelated docs(from )2Send prompts to

45、LLM to generate responseInstructionFollowingModel4Model ServingMonitoringChoose and load model(s)Model HubModels in Unity CatalogHuggingFace HubModels in MarketplaceAI ApplicationRAG/AgentChainQuery RAG modelModel Serving(CPU)UsersResponse51QueryETLPrepare docs(cleanse,chunk,)IngestdocsFilesTablesVo

46、lumesDelta Live TablesAutomatically sync with Delta table2Search for related contentData ServingVector SearchEmbeddingModelModel Serving(GPU)Compute embeddingsFeature Serving2024 Databricks Inc.All rights reservedArchitecture:fineArchitecture:fine-tuningtuning37Model HubAI Orchestrator&ToolsFilesLoa

47、d base model3Compute4myModelRegister customized model5Fine-tune modelModels in Unity CatalogIngesttrainingdocs1TablesGPU clusterPreparedata2VolumesData PreparationTensorFlowNotebooksTransformers SparkPyTorchDelta Live TablesModels in MarketplaceDeepSpeedMLflowRaySparkMosaic AI Model Training(Fine-tu

48、ning)TrainerPyTorch 2024 Databricks Inc.All rights reservedArchitecture:pretrainingArchitecture:pretraining38Model HubAI Orchestrator&ToolsFilesLoad model architecture3Compute4myModelRegister custom model5Pretrain modelModels in Unity CatalogIngesttrainingdocs1TablesGPU clusterPreparedata2VolumesDat

49、a PreparationTensorFlowNotebooksTransformers SparkPyTorchDelta Live TablesModels in MarketplaceMosaic AI Model Training(Pretraining)2024 Databricks Inc.All rights reservedPretraining a fully custom modelPretraining a fully custom model39Designed for enterprise useOpen-source for commercial useBase m

50、odel can be fine-tunedFast&accurate.For example,higher quality than Llama2-70B yet 2x faster for inference.From March 2024Foundation Model APIsAI PlaygroundDatabricks MarketplaceHugging Face Hub&GitHubDBRX:TopDBRX:Top-performing openperforming open-source,commercially viable LLMsource,commercially v

51、iable LLMIn terms of operations,In terms of operations,what did it take?what did it take?2024 Databricks Inc.All rights reservedPretraining a fully custom modelPretraining a fully custom model40It took Mosaic AI.It took Mosaic AI.Composer for optimized deep learning trainingStreaming Datasetfor effi

52、cient data loading during trainingLLM Foundryfor training,fine-tuning and evaluatingEvaluation Gauntletfor evaluating qualityNotebooksand Apache Spark for data cleaning and processingDelta Lake and Unity Catalog for data storage andgovernanceMosaic Multi-CloudTraining(MCT)totrain the modelMLflow and

53、 Lakeview forexperiment trackingFoundation Model APIsand AI Playground for eval and red-teaming2024 Databricks Inc.All rights reservedRecommendations on how to manage and deploy AI models safely and securely,by defining:12 AI system components55 technical AI risks53 mitigating controlsBuilt with ind

54、ustry luminaries,partners,and customers.Databricks AI Security FrameworkDatabricks AI Security Framework41Holistic approach to AI system securityHolistic approach to AI system securityWhitepaperDatabricks Security&Trust Center2024 Databricks Inc.All rights reservedGenAI at DAISGenAI at DAISProduct L

55、ed SessionsProduct Led SessionsTop Databricks product sessions:Mosaic AI Agent Framework/Quality LabWeds 11:20 AM-12:00 PM|South,Level 2,Rm 211Mosaic AI Vector SearchTuesday 9:00 AM-9:40 AM|South,Level 2,Rm 209Mosaic AI Model TrainingThursday 12:30 PM-1:10 PM|South,Level 2,Rm 211Mosaic AI Deep Dive+

56、Tools CatalogWeds 12:30 PM-1:10 PM|West,Level 2,Rm 2001Shutterstock ImageAI,Powered by DatabricksWeds 11:20 AM-12:00 PM|West,Level 2,Rm 2009Top Customer led sessions50 sessions.Recommended:JPMorganNorthwestern MutualCorningRolls-RoyceCVS HealthFoxDun&BradstreetComcastClick here to access all GenAI s

57、essions2024 Databricks Inc.All rights reservedGenAI at DAISGenAI at DAISGeneral RecommendationsGeneral RecommendationsTop Databricks led sessions 25 sessions.Recommended:Beginner:Introduction To Mosaic AI:How Databricks Simplifies Your Genai Journey Introduction To Retrieval Augmented Generation And

58、 Implementing With Databricks Introduction To Vector Search On DatabricksAdvanced:Deep Dive Into Building Production Quality Gen AI Applications Customizing Your Models:Rag,Fine-Tuning,And Pre-Training Deep Dive Into Mosaic AI:Getting Genai Apps To Production On Databricks How To Train Or Fine-Tune

59、A Custom Llm On Your Data With DatabricksTop Customer led sessions50 sessions.Recommended:JPMorganNorthwestern MutualCorningRolls-RoyceCVS HealthFoxDun&BradstreetComcastClick here to access all GenAI sessions2024 Databricks Inc.All rights reserved44LLMOps Q&ALLMOps Q&A2024 Databricks Inc.All rights reserved Read the Big Book of MLOps for more fundamentals and architecture Try out MLOps Stacks via the GitHub repo Try the RAG demo from the Databricks Demo CenterLearn moreLearn more45