Doris Summit-平安人壽基于 Apache Doris統一 OLAP 技術棧實踐-（脫敏版）.pdf

《Doris Summit-平安人壽基于 Apache Doris統一 OLAP 技術棧實踐-（脫敏版）.pdf》由會員分享，可在線閱讀，更多相關《Doris Summit-平安人壽基于 Apache Doris統一 OLAP 技術棧實踐-（脫敏版）.pdf（21頁珍藏版）》請在三個皮匠報告上搜索。

1、平安人壽基于 Apache Doris 統一 OLAP 技術棧實踐孫順平安人壽 大數據架構師Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit

2、 Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023目錄1 大數據平臺建設沿革與總覽2 早期架構與應用痛點3 基于 Apache Doris 統一 OLAP 技術棧的演進之路4 總結與未來規劃Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023

3、Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit A

4、sia 20231大數據平臺建設沿革與總覽Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit As

5、ia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023大數據平臺建設歷程20052022至今2009大數據產品體系數據倉庫與數據集市1.集中業務系統數據2.按需開發數據報表統一 OLAP 技術棧、統一指標與標簽管建、統一數據服務大數據 Hadoop 平臺1.存儲力和算力大2.提升報表刷新時效2016設計數據中臺1.數據治理2.數據底座3.數據產品Doris Summit Asia 2023Doris Summit A

6、sia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris

7、Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023大數據產品體系總覽Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summi

8、t Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 20232早期架構與應用痛點Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit As

9、ia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023早期架構總覽Doris Summit Asia 2023D

10、oris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit As

11、ia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023早期應用痛點報表分析HivePostgreSQL即席查詢HivePrestoPostgreSQL總部業務運營人員管理層經營數據探查痛點升級需求業務洞察問題定位趨勢預測經營全貌概覽多輪清洗計算，數據鏈路長PG 擴容成本高導數冗余、資源浪費應用層需配合改造大規模數據處理能力報表產出時效性報表查詢響應在百毫秒內資源成本控制客戶經營產品、理賠、核保等數據可視化分析指標數據集訂閱痛點查詢性能不及預期Hive 權限管理不清晰缺少資源隔離限制升級需求支持高并發查詢

12、支持低延遲響應指標復用率Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Do

13、ris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023早期應用痛點人群圈選ClickhouseKylin升級需求CK 難以支撐 200 人并發查詢組件語法差別，適配成本高依賴人工判斷、靈活度不足運維成本高多維分析Druid一線業務運營人員高并發查詢訪問報表查詢響應時間指標開發擴展性精確去重與關聯查詢痛點業務運營決策業績報表實時數據監控和分析節點重啟時間超 24 小時查詢靈活度低非精確去重運維成本高升級需求總部與分公司營銷人員人群圈選畫

14、像分析人群檢測與追蹤痛點支持高并發查詢多表關聯查詢能力數值間圈選Array 字段Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia

15、2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023早期數據開發系統痛點標簽數據開發組件 HBase指標數據開發組件 PG Presto Druid 1.平臺組件不同，數據未打通2.一個需求配置一個接口，無法動態配置與靈活調用痛點 依賴預計算 不同組件支持指標存儲與查詢 不支持復合指標開發痛點 不支持二級索引 日期、數字需額外轉化 不支持標簽跨對象Doris Summit As

16、ia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris S

17、ummit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023架構升級目標強化查詢性能豐富查詢與寫入的優化功能打通平臺數據讀取，將指標與標簽數據統一存儲，實現復合指標加工數據治理體系化、數據一致性覆蓋不同應用系統的日常分析需求，面向更多樣化的業務場景綜合性強、靈活度高有效簡化數據鏈路、架構技術棧，降低對技術人員開發的依賴開發、存儲、運維成本控制高效寫入與極速分析持續構建一站式數據門戶Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit

18、Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris

19、 Summit Asia 2023Doris Summit Asia 20233基于 Apache Doris 統一 OLAP 技術棧的演進之路Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Su

20、mmit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023為什么選擇 Apache Doris 高效的寫入性能極速的查詢分析簡單易用的架構 支持主鍵模型寫時合并 支持 Schema 在線變更 支持 Upsert 所有導入方式，條件更新、條件刪除、部分列更新以及分區覆蓋 支持多種存儲模型 支持物化視圖與 Rollup 預聚

21、合 支持高并發與低延遲查詢 支持多表關聯查詢 提供靈活的數值區間圈選 支持 Bitmap 精確去重，HyerLogLog 近似去重 采用 MySQL 協議，支持標準 SQL MPP 架構，僅 FE 與 BE 兩類進程 節點間負載均衡 列式存儲，極高壓縮比Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2

22、023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023基于 Apache Doris 構建全新大數據產品體系統一 OLAP 技術棧 Doris 集群之上構建統一的指標標簽設計平臺 形成“上下經營一張表”完善經

23、營指標追蹤體系統一的指標標簽設計平臺 統一的數據查詢與計算引擎 可插拔引擎 減少重復開發、冗余存儲 API 接口服務化 打通各類應用系統的數據調取 統一的數據服務平臺Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2

24、023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 202314倍10倍統一 OLAP 技術棧，加速業務需求交付原架構鏈路長、導數重復、定制開發14 個 PG 數據庫成本壓力業務交付耗時 2 周新架構 架構統一、運維成本降低豐富的數據模型簡化鏈路業務交付僅需 1-2 天原架構缺少權限、表交叉使用查詢分

25、鐘級耗時新架構業務人員只可以訪問 ADS 層數據提升數據指標復用率查詢秒級毫秒級響應查詢響應提速交付效率提升Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Dori

26、s Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023統一指標和標簽管理，直通應用場景指標和標簽服務 API指標與標簽定義指標與標簽統一計算數倉層引擎層應用層Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summ

27、it Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023統一

28、數據服務平臺，打破數據孤島Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023D

29、oris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 20234總結與未來規劃Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris S

30、ummit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023湖倉一體化基于 Doris 的統一分析架構Apache Doris湖倉（Hive，Hudi）之前的分析架構Hive湖倉HudiDruidOLAPPrestoClickhous

31、ePostgreSQLHBaseKylin更開放、靈活、可擴展的企業級管理與分析大數據產品體系Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summi

32、t Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023多租戶資源隔離Resource Group 機器分組 副本放置到分組 User 綁定分組WorkLoad Group 對特定查詢模式、分區/分片數大的查詢攔截 單查詢內存限制 多 Cluster 機制和 Resource Group 整合完善應用系統之間負載均衡性能，使集群資源更合理分配Doris Summit

33、Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris

34、 Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023獲取更多社區動態與最佳實踐Doris Summit 峰會官網：doris- Doris Summit 峰會回放：https:/ Doris 官網：doris.apache.orgApache Doris GitHub： Doris 官方平臺：Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Su

35、mmit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023