大數據AI一體化的解讀.pdf

1、PAI&Dataworks 大數據AI一體化的解讀AI “iPhone”統一管理資源算力運營生成式AI智能計算科學計算AI資源效率：分布式AI怎么跑AI人力效率：AI怎么寫智算平臺核心目標 提升2個效率+80%=20%ModelDataModel-centric AIModelDataData-centric AISource:A Chat with Andrew on MLOps:From Model-centric to Data-centric AlData-centric MLModel-centric MLWorking on code is the central objectiv

2、eWorking on data is the central objectiveOptimizing the model so it can deal with the noise in the dataRather than gathering more data,more investment is being made in data quality tools to work on noisy datainconsistent data labelsData consistency is keyData is fixed after standard preprocessingCod

3、e/algorithms are fixedModel is improved iterativelyIterated the data qualityAIODPS 大數據計算集群PAI-靈駿智能計算集群云原生通用計算集群-DataWorksPAI-AI智能推薦PAI-Rec開放搜索 OpenSearchPAI-APIPAI-PAI-靈積PAI-智碼實驗室-/-LlamaAI for ScienceODPSPAI-AIAIPaaS海量數據加工(SOL、Python)ODPS-MaxCompute交互式數據分析ODPS-HologresFlink結構化數據集成Hadoop數據遷移CPUGPURD

4、MACPFSOSSNAS EMR（Spark、StarRocks）DLFDLFElasticsearch生態模型服務PAI-EAS智算服務PAI-靈駿AI加速引擎PAI-ACC分布式訓練PAI-DLCDW數據建模DW數據開發DW數據治理DW數據質量DW數據安全交互式建模PAI-DSW可視化建模PAI-DesignerMLOpsPAI-OuickStart數據標注PAI-Itag特征工程PAI-FeatureStore1/MC/DW/PAI/Flink/Holo TextCHUNKQ&ACHUNKQ&AembeddingsEmbedding Model BEG/SGPT/text2vec PDF

5、TXT QA/PAIidcontentembeddingdoc_ididPAI0.1,-0.1,0.1PAIidmapjoin0.5,0.2,0.9MCidPAI0.8,-0.1,0.7PAIidHolo0.6,0.9-1.1HoloHolo/Elasticsearch/AnalyticDB/FAISS PAI+query LLM+SFT ChatGPT/Qwen/embeddingsEmbedding Model BEG/SGPT/text2vecPrompt Engineering/BladeLLMLLM+SFT ChatGPT/Qwen/+1,2,3PAILLM1.PAI 2.PAI 3

6、.PAICPU/GPU2用戶行為日志實時計算 Flink特征庫批流統一樣本生成（Flink）樣本庫批流統一樣模型訓練（PAI-TF）數據分析模型中心1n在線預測模型部署模型驗證離線計算 MaxCompute多種資源支持，AI+大數據PAI-Designer有效連接大數據+AI各種產品形態和資源ECS/EGSPAIMaxComputeFlink多種構建Flow的方式Pipeline SpecPaiFlow SDKGUIPai-DesignerEAS ServiceUserPipeline SpecHTTP APIHTTP APIPAIFlow ServicePipeline Run DAGJob

7、 ServiceDLC JobODPS JobFlink JobPAI-EASNodeNodeNodeNodeNodeOperator Registryconstruct DAGsubmitServerlessFrom“Gartner Reference.Architecture for Multi-tenancy”G00205983Multitenancy1 Shared Nothing2 Shared Hardware3 Shared OS4 Shared Database5 Shared Container6 Shared Everything7 Shared MultitenancyA

8、pplicationLogicApplicationPlatformDataPlatformManaged Virtual MachinesManaged OS ProcessesTenantTenantApp.App.APAPDPDPSys.Inf.Sys.Inf.TenantTenantApp.App.APAPDPDPSystem InfrastructureTenantTenantApp.App.APAPDPDPSystem InfrastructureTenantTenantApp.App.APAPCloud-EnabledData PlatformSystem Infrastruct

9、ureTenantTenantApplicationLogicCloud-EnabledApplication PlatformDPDPSystem InfrastructureTenantTenantApplicationLogicCloud-EnabledApplication PlatformCloud-EnabledData PlatformSystem InfrastructureTenantTenantCloud-EnabledApplication LogicApplication PlatformData PlatformSystem Infrastructure高技術復雜度低

10、低單位計算成本高高資源共享程度低SLOJavaAIKoordinator CNCFSLOOptimization for Microservice,Web APPs,AI,Big Data Workloads Kubernetes 原生調度能力Pod Pod Gang SchedulingCapacity SchedulingCPUGPUNPUVPC/RDMANAS/CPFSOSSHeterogeneous computing resource managementWorkloadsResource SchedulingResource Management schedule framewor

11、k kubernetes QoS K8S Master Container Container GuesKernelPodVolumePodVolume Pod網卡 網卡 Pod網卡 網卡 RAMRBACRRSARRSAKMSK8S Master RAM/RBAC/RRSA kata 3.0 KMSEBSNASVPCvRoutervSwitchACSWeb Spark PAI MaxCompute ACS SLO K8S APIQuota多級Quota調度，支持Quota Tree 多層級調度能力，支持資源快速識別和搶占，從而短時間確保高優先級任務資源。使得用戶能夠精細管理多種任務形態下的資源

12、使用。Root|Quota=100Quota-1Min=20,Max=40Quota-2Min=40,Max=80Quota-3Min=30,Max=50Quota-nMin=x,Max=yQuota-1.1Min=10,Max=20Quota-1.2Min=10,Max=50Quota-2.1Min=10,Max=60Quota-3.1Min=10,Max=40Quota-n.1Min=a,Max=bQuota-2.2Min=30,Max=80未使用的資源任意使用的資源承諾資源彈性資源共享資源4workerw0-w1-w2-w3-w0 RingAllReduce512 30%40%PSW-1

13、GPU機器-AGPU機器-BGPU機器-CGPU機器-DASW-1ASW-2W0W2W1W3隨機調度PSW-1GPU機器-AGPU機器-BGPU機器-CGPU機器-DASW-1ASW-2W0W2W1W3拓補順序感知調度MaxCompute 4.0 Data+AISQLPandasMarsMaxFrame DAG driver MaxCompute Notebook/AIMarsPAI-DLCPAIAIMarsSQLPAI-EAS MaxFrameBigMetaMaxC OSSsubDAGSQL/pandas/MarssubDAGMars/PAI-TF/PAI-PytorchsubDAGMars

14、/SQL/PAI-TF/PAI-Pytorch DatasetAcc全托管、面向機器學習的云原生AI數據集加速服務RDMARDMARDMA DataSetAccPAI模型類型業務模型相比于云上對象存儲的加速比文件存儲(極速型)PAI-DatasetAcc圖像分類RetNet50Swin-Transformer7.20X8.67X12.31X12.83X語音識別ofa-sysofa-sysWenet6.5X36.89X8.0X41.73X多模態42630675246512416641569138737.543112912336RESTNET50SWIN-TRANSFORMERWENETOFASY

15、SAI讀取訓練數據速率（訓練文件個數/SECONDS）對象存儲文件存儲PAI-DatasetAcc對于互聯網數據來說，文本重復的比例通常在20%-60%之間。文本規模越大，重復比例越高。減少哈希沖突，縮小熱點桶近似減少計算量，而不影響精度文本分詞Jieba/千問文本映射為向量MinHash尋找相似文本對MinHashLSH去除重復文本圖連通分量算法N-gramFlinkML上構建的算法-Data Deduplication from Google(2022/03)-Text Deduplication from BigCode(2023/05)-The RefinedWeb for Falco

16、n LLM(2023/06)-SlimPajama:A 627B token,cleaned and-deduplicated version of RedPajama(2023/06)有效連接大數據+AI各種產品形態和資源 BI BI+AI30%+數據開發與分析效率提升Prompt Engineering數據集成數據建模數據開發數據治理數據分析與服務DataWorksCopilotAI智能助手DataWorks一站式開發治理平臺SQL大模型一站式平臺沉淀領域知識AI加速數據開發效率開源DDL+NLSQL數據(Hive)中文通用對話數據有監督微調基礎開源大模型底座業務DDL+NLSQL數據(M

17、C)元數據數據模型數據血緣數據指標企業專屬領域知識DataWorksCopilot 全新發布表快捷查找/SQL生成/SQL改寫 SQL糾錯/SQL解釋與注釋對話式自然語言交互界面DataWorks Copilot:Facilitating Data Development*以上結果依據真實場景數據估算，最終效果以實際產品和場景測試數據為準DataWorksDataWorks AIDataWorks聯合DataV數據可視化，深度結合AI技術，幫助你講好數據故事，表達數據觀點結合大模型技術，通過自然語言指令，即可生成 SQL 查詢數據AI 智能數據查詢一鍵生成數據長圖報告，支持多種精美主題隨意切換，輕松打造個性化數據可視化作品，并支持一鍵分享一鍵構建和分享數據報告結合 AI 技術，數據圖表卡片自動生成，提供多樣化的可視化表現方式，幫助您快速獲得靈感、保存見解AI 自動數據圖表生成統一自動探查數據集，無需專業技術背景即可快速了解數據內容、統計分布自動數據探查AI大模型智能問答知識庫智能數倉智能數據治理智能搜索推薦數據湖數據倉庫數據處理分布式計算數據科學計算機器學習深度學習模型開發模型訓練模型推理AITHANKS