Doris Summit - 奇安信基于 Apache Doris 的日志安全分析系統升級實踐（脫敏）.pdf

1、奇安信基于 Apache Doris 的日志安全分析系統升級實踐舒鵬奇安信 服務端技術專家Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit

2、Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023目錄2 架構升級之旅3 Apache Doris 2.0 查詢提速實踐經驗4 規劃與展望1 公司介紹與業務背景Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asi

3、a 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 20231公司介紹與業務

4、背景Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit

5、 Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023奇安信科技集團股份有限公司中國企業級網絡安全市場的領軍者專注于為政府和企業用戶提供新一代網絡安全產品和服務。2023 年，奇安信在數據安全領域全面發力。目前核心產品天擎終端安全系統在國內已有 4000 萬政企用戶部署、全國部署服務器超過 100 萬臺、服務超 40 萬大型機構。作為網絡安全國家隊，奇安信立志為國家構建安全的網絡空間，在終端安全、云安全、威脅情報、態勢感知等領域的技術研發持續領先?！笆?/p>

6、五”規劃開局起步，數字化轉型全面鋪開推動網絡安全需求井噴，奇安信將繼續為政府與企業等用戶提供全面、有效的網絡安全解決方案，向成為“全球第一的網絡安全公司”的愿景目標不斷奮進Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asi

7、a 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023原日志安全分析系統輸入你的小標題外線框是文字輸入區參考線，請在正式稿中刪除。文字：根據文章長度選擇適合的文字尺寸，以28-24號字為宜。輸入你的小標題外線框是文字輸入區參考線，請在正式稿中刪除。文字：根據文章長度選擇適合的文字尺寸，以2

8、8-24號字為宜。Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris

9、 Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023架構主要痛點查詢性能寫入效率 原系統只能通過 SQL LIKE 進行全量掃描和暴力匹配。原系統不支持對文本字段，如 URL、payload 等關鍵字進行模糊匹配 千億級數據量的查詢耗時近分鐘級，當遇到并發查詢，性能還會進一步惡化 每天所生產的安全日志數據達到千億級，且每天新增日志量在不斷增長中。原架構系統入庫速率逐漸降低，集群寫入壓力過大、高峰期數據擠壓嚴重、穩定性造成影響 集群多次擴容

10、（3 節點至 13 節點），機器成本已超預期的情況下，寫入效率沒有得到改善Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 202

11、3Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023系統建設目標 海量病毒查殺事件實時寫入與存儲 基于日志數據 Schema Free 特性支持豐富數據類型的寫入與變更實時寫入性能 支持對字符串提供模糊查詢的能力 能夠靈活創建且類型豐富的索引，例如倒排索引以加速篩選過濾數據倒排索引&模糊查詢 系統自身的運維簡易程度 系統是否具備合適的管控工具簡易運維&高效管控工具日志數據價值密度低，

12、但存儲的規模很大、存儲周期相對較長高性價比存儲成本Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summi

13、t Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 20232架構升級之旅Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia

14、2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023為什么選擇 Apache Doris主要研究探索 OLAP 數據庫中擅長日志分析系統的組件，構建一體化日志存儲分析平臺，賦能網絡安全。選擇 Doris

15、的理由：1.支持標準 SQL 語法，與 MySQL 高度兼容；2.具備倒排索引、Ngram BloomFilter 索引等檢索特性；3.支持聚合、多表 Join、子查詢、窗口函數、UDF、物化視圖等功能；4.能夠在線毫秒級刪減字段、按需增減索引、按需更改類型；5.支持 Text、JSON、Array、Map、Variant 等多種數據類型；6.采用 ZSTD 壓縮算法優化存儲空間占用；Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia

16、2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023全新日志安全系統架構

17、Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit A

18、sia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023200%690%寫入性能提升寫入與查詢性能提升原系統寫入吞吐13臺服務器CPU 利用率 30%寫入吞吐 30 萬/條查詢性能提升Apache Doris 寫入吞吐3臺服務器CPU 利用率 100%寫入吞吐 108 萬/條原系統查詢耗時79 條 SQL 查詢語句整體耗時 1757.32sApache Doris 查詢耗時79 條 SQL 查詢語句整體耗時 253.12sDoris Summit Asia 2

19、023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summi

20、t Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023存儲成本降低40%存儲成本降低1.數據壓縮比 1:4.31:5.72.數據膨脹率 3-5 倍1 倍ZSTD 是一個優秀的新型壓縮算法，使用了智能優化算法，相較于常見的 GZIP 算法，ZSTD 具有更高的壓縮率更和更快的壓縮解壓速度，尤其在處理日志場景時表現非常出色。Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023D

21、oris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit As

22、ia 2023運維與管控成本降低Cluser Manager For Apache Doris由飛輪科技免費開放可視化集群管理工具滿足日常運維中集群監控、巡檢、修改配置、擴縮容、升級等操作。降低登陸機器手動操作的麻煩和誤操作風險?？梢暬罩咎剿鞣治?WebUI支持關鍵詞檢索、趨勢圖展示、趨勢圖拖拽日期范圍、明細日志平鋪和折疊展示、字段值過濾等交互方便的探索式分析。對于習慣 ELK 日志分析的用戶非常友好，契合日志場景探索下鉆分析需求。Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit As

23、ia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris S

24、ummit Asia 20234Apache Doris 2.0 查詢提速實踐經驗Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia

25、 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023SQL查詢對比所有查詢語句均有明顯提升，整體性能查詢提升近 7 倍，26 條 SQL 查詢語句性能提升 10 倍以上：線上去重分析 79 條 SQL，在同一天 1000 億條總數據、同樣 10 BE 節點集群規模上對比測試查詢耗時。8 條 SQL 查詢提升 10-20 倍14 條 SQL 查詢提升 20-50 倍4 條 SQ

26、L 查詢提升 50 倍以上其中差異最高達 88 倍，在原系統執行接近 1 分鐘，在 Apache Doris 中僅需不到 1 秒Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asi

27、a 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023倒排索引 對關鍵詞查找加速-例如 Q43 提升 88.2 倍 Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 202

28、3Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023NGram BloomFi

29、lter 索引 對 LIKE 加速-例如 Q75 提升 44.4 倍 Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023D

30、oris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023TopN 日志明細查詢優化-例如 Q22，提升50.3倍 Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Su

31、mmit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 20235規劃與展望Doris Summit Asia 2023Doris Summit Asi

32、a 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Su

33、mmit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023擴大 JSON 數據類型的應用引入 Variant 可變數據類型支持存儲任意結構的 JSON 數據支持字段個數與類型變化更靈活地定義特殊字符，更好地實現半結構數據 Schema Free 分析需求Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit

34、 Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023獲取更多社區動態與最佳實踐Doris Summit 峰會官網：doris- Doris Summ

35、it 峰會回放：https:/ Doris 官網：doris.apache.orgApache Doris GitHub： Doris 官方平臺：Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023