1. Apache Doris 在聯通多業務場景下的應用與實戰.pdf

1、Apache Doris 在聯通多業務場景下的應用與實戰田向陽聯通西部創新研究院 研發一部 數據研發專家Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris

2、Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023目錄2.業務場景3.OLAP 選型4.平臺架構1.背景介紹5.未來規劃Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris S

3、ummit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023公司簡介1Doris Summit Asi

4、a 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Su

5、mmit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023聯通西部創新研究院是中國聯通集團在西部地區布局的創新主體，由聯通數字科技有限公司出資設立，陜西省分公司協同運作，并與西安交通大學、西安電子科技大學等高校開展深層次聯合創研。聯通西部創新研究院將原數科公司總部西安研發中心和原云數據事業部西安研發中心整體納入，錨定“做成規模專業化高水平的研發機構”高定位，內部設立多個研發部門及技術部，分別承擔數科公司各事業部的重點研發項目及任務，肩負“先行者”“排頭兵”使命感，持續推動公司研發集約化。Dor

6、is Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia

7、 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023技術布局Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris

8、 Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023業務場景Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit As

9、ia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023豐富的業務場景

10、集群規模無配套小集群大集群超大集群多行業生產制造經營分析海量日志智慧政企.數據量GB級TB級PB級數據需求數據資源目錄可視化大屏自主分析海量數據檢索算法模型.Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023D

11、oris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023工業互聯網-5G全連接項目特點5G全連接平臺是助力生產制造企業數字化轉型的能力底座平臺，具備工業數據采集、工業數據治理、工業業務分析、低代碼大屏等能力（應用集成+數據集成）1.數據源類型豐富 主要來自于各個業務系統數據、物聯網設備數據、日志數據，采集

12、方式分為實時和離線批量采集2.業務訴求 數據需求主要是提供大屏、在線自主分析、經營報表3.數據量中等4.存儲方式 Doris+時序數據庫Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit

13、 Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 20235G 全連接功能架構Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asi

14、a 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 20235G 全連接Doris Summit Asia 2023Do

15、ris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asi

16、a 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 20235G 全連接數據架構Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 20

17、23Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023老架構離線數據鏈路Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Dor

18、is Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023老架構實時數據鏈路Doris Su

19、mmit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023

20、Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023業務痛點離線產出報表需要同步到MYSQL提供查詢實時鏈路計算復雜度高數據鏈路長且復雜實時計算采用多流join由于業務更新復雜性導致join結果正確性有3%-5%偏差，每天通過離線數據修正數據準確性問題寬表模型下，業務維度變更需要對歷史數據離線重刷維度變更問題整個架構涉及技術棧和組件較多，維護成本高維護成本高Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit A

21、sia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris

22、Summit Asia 2023Doris Summit Asia 2023新架構Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia

23、 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023基于 Doris 新架構的收益老架構完全依賴于hadoop生態，新的架構可根據客戶的數據規模對hadoop生態進行取舍，架構上有了更靈活的選擇架構靈活性提高老架構olap查詢使用了寬表模型，固化了維度，而新的架構直接將dwd事實表和維表開放給應用，直接實現olap查詢，提供給業務更高的靈活性業務靈活性準確性提高老架構在基于h

24、ive數據建模加工之后將數據同步到clickhouse提供給應用查詢，新架構直接doris通過Doris提供查詢，避免了額外的數據同步鏈路數據處理鏈路縮短相比于之前老架構的小時級數據延遲，新的架構支持數據寫入秒級延遲。且實現了流批一體，計算結果的一執行數據時效性、準確性（實時場景）提高Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Dor

25、is Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 20235G 全連接應用1.70%的核心業務實現準實時（核心業務Flink CDC+Doris）實時鏈路+(邊緣業務JDBC ca

26、talog)定時批量抽取2.報表毫秒級響應 JDBC+Doris 維度建模定時報表數據產出3.在線近實時分析多維分析 DWD明細表基于時間分區+分桶join實現業務效果Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia

27、2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023智慧政企-數字鄉村項目特點1.數據源類型單一主要來自于各個業務系統數據庫采集，采集方式分為實時和離線批量采集2.業務訴求數據需求主要是提供大屏、可視化報表、數據服務3.數據量小4.存儲方式：DorisDoris Summit Asia 20

28、23Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit

29、 Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023數字鄉村可視化大屏1.大屏指標毫秒級返回基于Doris物化視圖加速大屏指標查詢2.用戶中心讀寫分離Seatunnel Zeta+MYSQL CDC將用戶數據實時同步到Doris，提供數據讀取服務，實現讀寫分離。3.生成數據資產目錄平臺側根據數據業務域對數據表統一分類形成數據資源目錄，通過數據服務模塊對外提供數據開放能力實現業務效果Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit A

30、sia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris

31、Summit Asia 2023Doris Summit Asia 2023建模分析態勢大屏通過多種維度不同監控指標的組合，例如安全事件TOP5等。密切監控當前網絡安全態勢狀況，呈現出攻擊威脅的主要分布。追蹤溯源基于網絡日志數據和告警數據進行規則或智能挖掘，發現潛在的安全事件，例如釣魚郵件、非法訪問等，實現定向威脅感知。通過對安全事件的快速研判，還原整個攻擊鏈條，進行精準的溯源取證，保障網絡和數據安全。海量日志場景-安全數據分析平臺Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asi

32、a 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Su

33、mmit Asia 2023安全數據分析平臺業務需求 明細日志秒級查詢 數據聚合快速響應 SQL支持 實時日志，秒級實時落地 正則匹配查詢 JSON數據存儲和查詢 ARRAY數據統計計算Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Sum

34、mit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023安全數據分析平臺架構項目特點數據源內外部告警防火墻日志登錄日志操作日志.數據采集RyslogFlumeFlinkCDC實時數倉FlinkODSHDFSDWKafka數據應用MySQLApache DolphinSchedu

35、lerKafkaDoris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris

36、Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023安全數據分析平臺數據規模30T150億180T900億單表最大數據條數(20天)單表最大數據量(20天)每日入庫數據條數每日入庫數據量Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2

37、023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 20232OLAP 選型Doris Summit Asia 2023Dor

38、is Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia

39、 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023OLAP 選型對比引擎運維成本實時寫入實時更新低延遲查詢 多表關聯查詢擴展性強依賴HadoopDoris低支持支持是好好否ClickHouse高支持不支持是不好好否Hive中支持不支持否好好是數據湖方案中支持支持否好好是Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Sum

40、mit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 20233平臺架構Doris Summit Asia

41、2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summ

42、it Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Do

43、ris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asi

44、a 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 20234架構優勢Doris Summit Asia 2023Dor

45、is Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia

46、 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023基于Doris架構的優勢運維成本基于純 Doris 生態的輕量級的存儲架構，代替了之前的基于 Hadoop 的 Lamda 架構，實現運維成本降低超20%。開發效率90%的實時計算，基于 Doris 內部數倉加工體系，代替老的 Flink 多流 join 的復雜場景任務開發，實現開發效率提升超20%。數據準確率及時性基于 Doris 流批一體數倉建設，重點業務實現 cdc 數據整庫高效接入。實現數據準確率及時性提升超5%。交付周期開發效率提升，運維成本降

47、低帶來項目交付周期縮減超20%。Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 20

48、23Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 20235未來規劃Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris S

49、ummit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023簡化運維Doris Manager查詢優化深入優化探索下一步規劃嘗鮮Doris 2.0升級版本到2.0，新版引擎下高性能查詢體驗存算分離基于 Doris 存算分離架構探索探索

50、HDFS湖存儲架構下doris計算節點部署，聯邦查詢能力基于 Doris Manager 進一步簡化運維工作根據業務特點進一步優化，比如索引加速，慢查詢監控，資源隊列劃分等等Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit As

51、ia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023獲取更多社區動態與最佳實踐Doris Summit 峰會官網：doris- Doris Summit 峰會回放：https:/ Doris 官網：doris.apache.orgApache Doris GitHub： Doris 官

52、方平臺：Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023