SelectDB - 基于 Apache Doris 的現代化實時數據倉庫.pdf

1、SelectDB 基于 Apache Doris 的現代化實時數據倉庫馬如悅輪科技 CEO&Apache Doris PMCDoris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia

2、2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023SelectDB：基于 Apache Doris的現代化實時數據倉庫數據的實時導入與實時存儲實時數據多種查詢負載上都擁有極速性能極速性能彈性架構實現資源靈活擴展彈性架構開放設計支撐高效系統集成開放集成Doris Summit Asia 2023Doris Summit Asia 20

3、23Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit

4、 Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023SelectDB 提供兩種產品形態管理的 Apache Doris 企業分發版。部署在物理機/虛擬機、K8s 或者公有云/私有云上。SelectDB Enterprise全托管的 Apache Doris 公有云服務。已經上線阿云、AWS、華為云和騰訊云。SelectDB CloudDoris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asi

5、a 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023SelectDB

6、 發展歷程2022年H22022年H22023年H12023年H2SelectDB Enterprise V3企業版SelectDB CloudV1SAAS 版本上線邀測存算分離架構阿里云、AWS、騰訊云、華為云SelectDB CloudV2增加可視化開發工具 WebUI配置無套餐化，改為僅需要指定vCPU數即可支持手動彈性伸縮、定時彈性伸縮和自動啟停。無感升級、緩存預熱SelectDB Cloud V3全面開放，達到GA內核更新至基于 Apache Doris 2.0BYOC產品形態Saving Plan上線GCP、亞馬遜云科技企業級內核：內核更新至 Doris 2.x企業級特性：跨集群復

7、制（CCR）可視化管控：Profile優化、Agent改造、日志查看、導入與集成、K8s部署支持數據遷移工具SelectDB Enterprise V2企業級內核：內核基于 Doris 1.2內置可視化開發 WebUI可視化管控：集群管理、監控報警、巡檢等SelectDB Enterprise V1企業級內核：內核基于 Doris 1.1命令行管控：集群創建、刪除、升級、擴縮容云服務Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2

8、023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023SelectDB云服務

9、SelectDB CloudDoris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023

10、Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023SelectDB Cloud 存算分離架構帶來極致性價比共享對象存儲與本地高速緩存多計算集群，實現負載隔離集群計算節點彈性擴縮容（手動、分時、自動啟停）云服務層應用系統/客戶端讀寫讀寫倉庫1（Warehouse 1）倉庫2（Warehouse 2）對象存儲（S3/OSS/COS/OBS）對象存儲（S3/OSS/COS/OBS）讀寫讀寫讀寫讀寫讀寫讀寫集群2（Cluster 2）

11、計算緩存（vCPU，RAM）（Cache）集群1（Cluster 1）計算緩存（vCPU，RAM）（Cache）集群3（Cluster 3）計算緩存（vCPU，RAM）（Cache）集群4（Cluster 4）計算緩存（vCPU，RAM）（Cache）集群5（Cluster 5）計算緩存（vCPU，RAM）（Cache）集群6（Cluster 6）計算緩存（vCPU，RAM）（Cache）計算集群層共享存儲層存算分離與彈性計算存儲冷熱分離，冷數據使用低成本可靠對象存儲，平均帶來 5x 左右存儲成本節省計算節點通過彈性擴縮容，減少計算資源浪費，平均帶來 5x 左右計算成本節省極致性價比Doris

12、 Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2

13、023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023無套餐 Serverless化Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia

14、2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023彈性擴縮容：手動、定時、自動啟停Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asi

15、a 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Su

16、mmit Asia 2023SelectDB Cloud 降低運維復雜度，提升開發效率多云統一管理組織管理、計費管理、云市場打通集群管理：Serverless化配置，無需配置節點套餐與數量安全連接：支持公網連接及 IP 白名單設定；支持私網連接監控與報警可視化篇理控制臺，隆低運維復雜度數據查詢數據管理可視化開發 WebUI，提升開發效率數據集成權限管理Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit

17、Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023SelectDB Cloud 兩種部署形態全托管控制面

18、和數據面在 SelectDB 賬戶最小管理更加 Serverless 化SelectDB Cloud SaaS全托管控制面在 SelectDB 賬戶，數據面在客戶賬戶數據合規要求比較高擁有較高云資源折扣和自己賬戶的其他系統方便的打通SelectDB Cloud BYOCDoris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summi

19、t Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023SelectDB Cloud 全面開放，免費試用國際站已經上線：即將上線：中國站已經上線：即將上線：Doris Summit Asia 2

20、023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summi

21、t Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023阿里云 SelectDB 版本邀測版上線Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris

22、 Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023SelectDB企業版SelectDB EnterpriseDoris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summi

23、t Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Dor

24、is Summit Asia 2023Enterprise Core：與 Apache Doris 100%兼容的企業版內核飛輪科技嚴格測試的穩定內核比社區更快速的缺陷修復 提供對大版本的長周期支持 長周期支持的穩定內核數據查詢數據管理數據集成權限管理內置可視化開發 WebUIDoris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris S

25、ummit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Enterprise Manager:可視化集群篇控工具可以管理開源 Doris 內核和企業版內核支持托管已部署集群支持多集群的創

26、建、配置變更、升級、擴縮容等功能提供監控、報警、巡檢、審計等功能支持在物理機/虛擬機、K8S 部署(開發中）和公有云部署(開發中）支持對接客戶的私有云降低運維復雜度，提升運維效率Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit

27、Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Enterprise Support：飛輪科技的專家技術服務月 Top 15 貢獻者，12 名在飛輪科技消除風險：提供例行巡檢，及時消除系統可能的隱患。專業團隊，斷絕一切后顧之憂解決問題：嚴格的服務SLA，確保 7*24小時及時獲

28、得專屬支持；對緊急Bug提供天級別的專屬修復版本更新。咨詢培訓：提供對業內最佳實踐的咨詢、產品的系統培訓。優化系統：通過對產品的使用培訓，業內最佳應用實踐的分享，與客戶一起優化系統性能和成本。5 名 PMC 成員，21 名 Committer主導發布了 Apache Doris V1.0 V1.1 V1.2 V2.0 等一系列版本Apache Doris 社區用戶群 95%的問題由飛輪科技同學響應最近兩年，Apache Doris 90%的新增代碼由飛輪科技同學貢獻Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 202

29、3Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit

30、Asia 2023Doris Summit Asia 2023自主可控、安全可靠多項國產化信創兼容認證公安部等保三級，6 項 ISO 安全管理體系認證，2022 信創產業實者Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit As

31、ia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023SelectDB解決方案SelectDB SolutionsDoris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 202

32、3Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit

33、Asia 2023SelectDB 的主要應用場景構建用戶數據平臺，進行用戶參與、留存和轉化等行為分析，及人群洞察和人群圈選等畫像分析 用戶行為與畫像分析構建統一的日志管理與分析平臺，在極低成本存儲的基礎上提供高性能的日志檢索與分析日志管理與分析面向內部和外部的實時報表與儀表盤，面向自動化程序的實時決策實時報表與實時決策對 TB和PB 數據進行快速的即席查詢(Ad-hoc Queries)交互式探索分析無論是混合部署(公有云、私有化部署)還是在多個公共云上部署，統一體驗，一切可簡化混合云和多云部署Doris Summit Asia 2023Doris Summit Asia 2023Doris

34、 Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2

35、023Doris Summit Asia 2023Doris Summit Asia 2023混合云和多云正在成為用云主流趨勢開源開放，與 Apache Doris 100%兼容。提供可以在 IDC、私有云部署的 SelectDB 企業版。提供在全球七個主流云上云原生的 SelectDB 云服務。所有形態都擁有一致的使用體驗。靈活的產品使用和部署形態SelectDB Cloud 提供可視化管控，對多云上的倉庫和集群進行統一納管。SelectDB Enterprise 提供可以私有化部署的 Enterprise Manager，可以對部署在物理機/虛擬機、K8s 容器和公有云/私有云上的集群進行

36、統一納管。統一的可視化管控工具提供跨集群復制能力，可以在多云上進行集群異步復制。提供數據遷移工具，方便從私有化部署遷移到云上，方便在云與云之間遷移。數據貫通和遷移Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 202

37、3Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023實時報表與實時決策支持 100 萬行/秒的實時寫入，也支持流式的從 OLTP 數據庫和 Kafka 同步數據。支持數據更新，關系數據庫中某些數據，例如訂單狀態等，可以實時地反映到數倉中，及時生成有關訂單狀態等信息的分析報告。實時數據寫入存儲和計算

38、引擎全鏈路向量化，實現數量級的查詢加速。強一致聚合物化視圖，自動使用預聚合結果，滿足亞秒級聚合統計查詢。亞秒級查詢響應采用分區分桶將單個查詢分發到單臺機器處理，使用跳數索引(Zonemap,Bloomfilter)和點查索引(主鍵、倒排)減少讀取數據量，提升并發查詢能力。通過點查行存，減少 IOPS 壓力，支撐單機上萬 QPS 的高并發點查。高并發的解決方案無單點故障，在線擴容、升級自動副本均衡，實現單集群高可用?？缂簩崟r數據同步，實現跨集群高可用。穩定、高可用的服務Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2

39、023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summi

40、t Asia 2023Doris Summit Asia 2023從固定報表到交互式即席分析SelectDB 支持通過 multi-catalog 的機制將外部的數據庫、數據湖映射為 SelectDB的庫表外表和內表可以聯邦查詢分析。做到了無需移動數據，即可使用 SelectDB 完成多各種數據源的即席查詢分析。聯邦查詢極速查詢性能：基于成本的優化器，MPP的分布式執行框架，向量化的計算引擎等對多種查詢負載提供極速查詢性能：不論是簡單的大寬表查詢，還是復雜的多表 JOIN 查詢，都擁有極速性能。多種查詢負裁上的極速性能SelectDB Cloud 提供多計算集群能力，Ad-hoc 分析集群和在

41、線服務查詢集群共享一份存儲數據的同時，做到了集群隔離。計算集群內，可以劃分不同的查詢資源組，來隔離不同用戶的查詢。異常 SQL 的捕獲：及時發現并 Kill 可能會造成節點內存 OOM 的查詢。內存消耗較大的 SQL，會對中間結果進行落盤，減少對內存的消耗。多計算集群與集群內的負載隔離Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Dor

42、is Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023用戶畫像與行為分析支持行為分析業務屬性頻繁變動，支持畫像場景的寬表列實時更新。毫秒級加列、部分列更新豐富的行為分析函數，帶

43、來開發簡化和效率提升，比如 rentention、window_funnel、sequence_match 等。函數持，業務場景秒級響應表正交位圖實現畫像場景的秒級圈用戶畫像千億數據，秒級人群預估，秒級別 10 標簽圈人，10 秒級別 100 標簽人群圈選?；?SelectDB 建設的 CDP 平臺Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia

44、 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023日志管理與分析為日志場景優化的高性能倒排索引，寫入速度是 ES 倒排索引的 4 倍。服務端 group c

45、ommit 機制，保證秒級實時可見前提下寫入吞吐到 GB/s。海量數據實時寫入采用列式存儲、精簡倒排索引、高壓縮比壓縮，存儲空間占用僅需 ES 的 1/5。采用冷熱分離，單位存儲空間成本僅需 ES 的 1/3，整體成本 1/15。高性價比存儲高效處理業務場景通過分區分桶、時間段篩選，大幅縮小查詢的數據范圍。通過倒排索引，按檢索詞快速定位到匹配的日志行，避免大批量掃描，達到秒級響應。Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 20

46、23Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023謝 謝 觀 看Thank

47、 Youhttps:/ Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Do

48、ris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023獲取更多社區動態與最佳實踐Doris Summit 峰會官網：doris- Doris Summit 峰會回放：https:/ Doris 官網：doris.apache.orgApache Doris GitHub： Doris 官方平臺：Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris

49、 Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023