9. 基于 Apache Doris 構建多場景極速分析體驗.pdf

1、基于 Apache Doris 構建多場景極速分析體驗王天宜飛輪科技 資深解決方案架構師Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit A

2、sia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023目錄2.Apache Doris 核心能力與特點3.Apache Doris 多場景實時分析方案1.Apache Doris 在大數據生態中的定位Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 202

3、3Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit

4、Asia 20231Apache Doris在大數據生態中的定位Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Dori

5、s Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Apache Doris 在數據分析中的定位CDCFlinkSparkRDBMSDTSLOGKafkaIOT報表分析即席分析湖倉聯邦分析日志檢索分析ETL/ELT高并發數據服務大屏，駕駛艙自助 BI 平臺訂單、運單分析廣告、營銷分析用戶行為分析AB 實驗平臺日志分析（替 ES）時序數據分析統一湖倉平臺SourceIntegrationData

6、LakeApplicationApache DorisHiveIcebergHudiDoris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asi

7、a 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Apache Doris 主要特性統一 結構化與半結構化統一 在線與離線數據統一 湖倉統一 多元數據建模統一簡單高效 極速的分析性能 高效的數據更新能力 豐富的數據集成能力 極致彈性與存算分離 高可用與高可靠 多租戶管理能力 可視化管理工具 豐富周邊工具Doris Summit Asia 2023Doris Summit

8、Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris

9、 Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 20232Apache Doris實時AdHoc分析場景Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia

10、 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023實時報表痛點問題ODS 源數據層DWD 明細層DWM 匯總層DWS 主題層數據質量管理數據入庫及更新問題數據導入時效性要求高，秒級別數據可見基于上游 TP 庫的主鍵需要實時更新數據需要不丟不重，重要數據原子性導入數據查詢時效性問題使用人數

11、眾多，并發量告查詢延遲要求高，延遲控制在秒級別基于場景不同，要求多樣的建模方式Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2

12、023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Apache Doris 高性能更新能力MemTableKeyC1C2KeyC1C210000001V6V7KeyC1C200010010V6V8KeyC1C210000001V7V8KeyC1C20001V8SegmentDeleteBitmapRowSet1version 0-5RowSet3version 7-7RowSe

13、t2version 6-6RowSet3version 8-81.Flush to File2.Lookup rowkey in batch,Then update the bitmap forcurrent versionMerge on Write 模型Merge on Write 更新能力實現原理：主鍵索引+Delete Bitmap 實現導入過程成數據刪除標記 delete bitmap基于 mow 模型實現查詢謂詞下推適用場景：適用于小批量實時高頻導入，基于主鍵做高頻數據更新支持 UPSERT、條件更新、條件刪除、部分列更新、分區覆蓋100 并發無 Cache有 CacheMoR23

14、.314.3MoW5.60.3提升倍數4.2x47.6xTPCH 標準測試集，使用 MOW 模型，性能提升近 50 倍Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 202

15、3Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Apache Doris 高性能更新能力游戲行為分析場景存量數據 300 億，單副本 80TB、70字段15 并發 Upsert，TPS 40w/s游戲行為分析場景半年物流訂單分析，200 字段寬表8 并發 Upsert，TPS 6w/s游戲行為分析場景千億數據月度統計，萬億數據年度統計1

16、0 并發 Upsert，TPS 10w/s消費金融場景寬表更新場景，基于 MOW 的部分列更新200 列，20 并發，每個并發更新 10 列數據可見性明顯降低，30s-10s某客戶 PoC 壓力測試40 并發 Upsert，10s CheckpointMOW 表穩定導數，TPS 7.5w/sDoris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023

17、Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Apache Doris 數據建模能力CREATE TABLE doris_agg_tab(store_id INT

18、,date INT,amount BIGINT SUM DEFAULT 0,)AGGREGATE KEY(store_id,date)DISTRIBUTED BY HASH(store_id)BUCKETS 32store_iddateamount102-142200202-151500402-154000store_iddateamount102-142000202-151500store_iddateamount102-14200402-154000原表新增結果AGGREGATE KEY報表統計、指標計算order_iddatestatus102-14完成支付202-15完成支付order

19、_iddatestatus102-14待支付202-15完成支付order_iddatestatus102-14完成支付原表新增結果CREATE TABLE doris_uni_tab(order_id INT,date INT,status VARCHAR(20)SUM DEFAULT 0,)UNIQUE KEY(order_id)DISTRIBUTED BY HASH(order_id)BUCKETS 32UNIQUE KEY訂單狀態、用戶狀態user_iddateaction102-14瀏覽202-15收藏user_iddateaction102-16購買user_iddateactio

20、n102-14瀏覽202-15收藏102-16購買原表新增結果CREATE TABLE doris_dup_tab(user_id INT,date INT,action VARCHAR(60)SUM DEFAULT 0,)UNIQUE KEY(order_id)DISTRIBUTED BY HASH(order_id)BUCKETS 32DUPLICATE KEY明細數據、行為數據Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2

21、023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Apache Dori

22、s 極速分析能力100 并發V1.1.5V1.2.4V2.0.0(倒排)SQL1 平均執行時間0.1300.1400.023SQL1 平均執行時間1.1780.3970.048SQL1 平均執行時間0.6530.2840.028SQL1 平均執行時間4.6010.9720.055SUM(單位ms)6.5621.7930.154100110021003100410051006date201202203204205206storeX01X02X03X04X05X06prodCUACUBCUCCUDCUECUFcustdate:1001,store:201,prod:X01.date:1002,st

23、ore:202,prod:X02.date:1003,store:203,prod:X03.date:1004,store:204,prod:X04.date:1005,store:205,prod:X05.date:1006,store:206,prod:X06.行存Storage EngineBackendFrontendSQL ParserAnalyzerShort-Circuit PlanRPCSQL1 cache short-circuit plan1SQLN cache short-circuit planNPrepareStatement Map高并發查詢痛點：以點查 SELEC

24、T*FROM user_table WHERE id=xxx 為例寬表帶來 IOPS 放大的問題執行引擎和優化器對于點查操作過重SQL 解析規劃由 FE 負責，高并發易形成瓶頸Apache Doris 面向高并發數據服務優化方案：引入行列混存，解決 IOPS 瓶頸引入 PrepareStatement，解決解析瓶頸規劃點查短路徑優化，減少框架開銷高并發數據服務某手機廠商實時行報表查詢場景，使用倒排索引，性能提升 45 倍Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023

25、Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit A

26、sia 2023Apache Doris 解決方案收益極致查詢性能向量化 MPP 查詢Pipeline 執行引擎多維度分析多表物化視圖基于 AGG 模型的預聚合不丟不重事務性導入 2PC數據多副本存儲高并發查詢分區分桶剪裁行存儲加速某頭部短視頻公司面向廣告主在線報表平臺，2000 臺高配物理機，并發 QPS 數萬每秒。Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Su

27、mmit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 20233Apache Doris用戶行為分析及人群圈選Doris Summit Asia 20

28、23Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit

29、 Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023用戶行為分析業務特點行為數據屬性數據交易數據事件分析漏斗分析留存分析歸因分析路徑分析分布分析人群預估人群洞察人群圈選人群查看A/B 測試產品迭代精細化運營人群吸 引 力轉化路徑中斷路徑畫像數據采集與處理數據存儲與分析數據應用行為分析標簽結果事件畫像業務特點：標簽生成時間不統一，有實時更新需求快速圈選、生成人群包及 TGI 計算行為分析業務特點：業務屬性頻繁變動SQL 復雜，效率低數據量大，交互式分析，要求秒級延遲Doris Summit Asi

30、a 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Su

31、mmit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023畫像和行為分析 Apache Doris 解決方案實時數據同步離線數據同步活躍標簽行為標簽消費標簽會員標簽模型標簽活躍標簽會員維度行為變更移動埋點PC 埋點消費數據會員數據分析函數正交為圖管理查詢存儲存儲A/B 測試產品迭代精細化運營人 群轉化路徑中斷路徑人 群MQADSDWODS數據變更表結構變更Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Dori

32、s Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia

33、2023Doris Summit Asia 2023豐富的行為分析函數Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023D

34、oris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Apache Doris 極速分析能力倒排索引CREATE INDEX idx_request ON httplogs(request)USING INVERTED;-search for request contains word login SELECT*FROM httplogs WHERE request MATCH login

35、;-search for request contains word error and error SELECT*FROM httplogs WHERE request MATCH_ALL login error;倒排索引實現多維度快速檢索持定義分詞，實現全檢索，加速志場景100 并發V1.1.5V1.2.3V2.0.0(倒排)SQL1 平均執行時間175519906011SQL1 平均執行時間218610763SQL1 平均執行時間68715821127SQL1 平均執行時間6611181977SUM(單位ms)190074366740862Doris Summit Asia 2023Do

36、ris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asi

37、a 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Apache Doris 部分列更新能力uidnameagetag1tag2tagN主鍵活躍標簽StreamLoad 部分列更新：curl -location-trusted-u root:-H partial_columns:true-H column_separator:,-H columns:uid,tag2-T/tmp/update.csv http:/127.0.0.1:48037/api/db1/user_profile/_stream_l

38、oadInsertInto 部分列更新：SET enable_unique_key_partial_update=true;INSERT INTO user_profile(uid,tag2)values(1,xxx);Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit A

39、sia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Apache Doris 數據建模能力CREATE TABLE advertiser_view_record(click_timeDATE,advertiser VARCHAR(60),c

40、hannel VARCHAR(20),user_id INT)DISTRIBUTED BY HASH(click_time)BUCKETS 32;CREATE MATERIALIZED VIEW advertiser_uv ASSELECTadvertiser,channel,bitmap_union(to_bitmap(user_id)FROMadvertiser_view_recordGROUP BYadvertiser,channel;基于物化視圖的 UV 統計Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 20

41、23Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit

42、 Asia 2023Doris Summit Asia 2023Apache Doris 標簽寬表CREATE TABLE orders_based_wide(user_id INT,tag_dateDATE,age INT,gender CHAR(2),city VARCHAR(50),region VARCHAR(50),tag1 BIGINT,tag2 BIGINT,tag3 BIGINT,tag5 BIGINT,.tagN BIGINT)UNIQUE KEY(user_id)DISTRIBUTED BY HASH(user_id)BUCKETS 64;寬表圈人：SELECT user_

43、idFROM orders_based_table_wideWHERE tag1=23 AND tag2=18 AND tag3=27;寬表人群預估：SELECT COUNT(user_id)FROM orders_based_table_wideWHERE tag1=23 AND tag2=18 AND tag3=27;Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris

44、Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Apache Doris 標簽高表CREATE TABLE user_tag_hig

45、ht_bitmap(tag VARCHAR(50),tag_valVARCHAR(50),date_timeDATE,user_idsBITMAP BITMAP_UNION)AGGREGATE KEY(tag,tag_val,date_time)DISTRIBUTED BY HASH(tag)BUCKETS 8;判斷用戶是否在人群包內：SELECTbitmap_contains(user_ids,13643)FROMuser_segmentsWHEREseg_name=seg_1;高表圈人：SELECT orthogonal_bitmap_intersect_count(user_ids,ta

46、g,tag1,tag2,.)FROMuser_tag_bitmapWHEREtag IN(tag1,tag2);Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Do

47、ris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Apache Doris 高表寬表結合圈人WITH user_segment_act AS(SELECT orthogonal_bitmap_intersect(user_ids,tag,tag01,tag02,.,tag102)FROMuser_tag_bitmap WHEREtag IN(t

48、ag01,tag02,.,tag102)SELECTuser_id,age,gender,city,region,phone,.FROMorders_based_table_wide t0JOIN(SELECT uid FROM user_tag_bitmapLATERAL VIEW explode_bitmap(user_ids)tmp AS uid)as t1ONt0.user_id=t1.uidSTEP1：高表通過正交函數圈人STEP2：位圖展開后行轉列STEP3：高表結果 join 寬表結果Doris Summit Asia 2023Doris Summit Asia 2023Dori

49、s Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia

50、2023Doris Summit Asia 2023Doris Summit Asia 2023Apache Doris 解決方案收益用戶畫像千億數據，秒級人群預估，秒級 10 標簽圈人，10 秒 100 標簽圈人多維度建模寬表圈人Bitmap 高表圈人豐富分析函數留存、漏斗、歸因函數Bitmap 正交函數高性能導入活躍標簽部分列更新多數據源導入方案Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit A

51、sia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 20234Apache Doris綜合湖倉統一方案Doris S

52、ummit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 202

53、3Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023數據湖與數據倉庫能力數據倉庫能力數據湖能力 開放的生態 靈活的數據訪問方式 可擴展性 高性價比 高數據質量 極致查詢性能 數據治理及模型分析 高實時性Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Dori

54、s Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Apache Doris 湖倉一體方案主要應用場景湖倉查詢加速數據導入和集成統一

55、查詢網關ETL/ELT 加速，寫回開放湖倉存儲格式便捷的元數據和數據打通元數據映射、cache 和 自動刷新支持幾乎所有開放湖倉格式 和 meta store支持 ES 和關系型數據庫，并且插件擴展支持外表的認證鑒權，如 keberos,ranger分析加速利用 Doris 高效的分析引擎加速熱數據 Cache 到本地支持彈性計算節點，實現計算彈性加速外表處理結果可寫入內表，形成加速視圖湖倉一體架構Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summi

56、t Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023基于

57、DataLake 的實時業務改造ODSDWDDWSADSKafkaODSDWDDWSADSODSDWDDWSADSVIEWVIEWVIEW行為數據業務數據系統日志爬蟲數據實時分析實時大屏實時推薦實時查詢湖倉一體方案數據下沉到 Hive、Iceberg、Hudi 等湖產品中Doris 作為計算引擎，進行查詢加速離線數據入庫方案創建 Catalog 鏈接INSERT INTO doris_tab SELECT*FROM Catalog_tab分析加速用戶生產環境Hive 查詢 56minDoris Hive Catalog 加速值 7min通過 Doris 內表查詢 30sDoris 實時數倉改造

58、方案Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit

59、 Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Apache Doris 存算分離架構BECACHEBECACHEBECACHECluster 1BECACHEBECACHEBECACHECluster 2DISKFEMETAFEDISKMETAFEDISKMETAS3/OSS/HDFS/MinIO元數據服務層計算集群層共享存儲層FEDISKFEDISKFEDISKFEDISKFEDISKFEDISKFEDISK存算一體架構輪科技基于 Apache

60、 Doris 實現了存算分離模式的 SelectDB CloudSelectDB Cloud 將在 V2.1 貢獻給 Apache Doris存儲資源和計算資源分離，各彈性，更極致性價依賴夠穩定、吞吐的共享存儲，通常公有云上才有可以通過多 cluster 機制實現負載隔離，讀寫分離等機制存算分離架構簡單易部署，易運維，適合絕多數FE 和 BE節點都可以靈活擴縮容存儲持冷熱數據分層，將冷存儲下沉到對象存儲或HDFS可以持彈性計算節點，快速實現計算彈性Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Su

61、mmit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023

62、Doris Summit Asia 2023Apache Doris 解決方案收益Hive 查詢 56min，Doris Hive Catalog 加速值 7min，通過 Doris 內表查詢 30s彈性擴展高性能存算分離方案計算層彈性可擴展簡單部署無需數據遷移簡化導入流程統一查詢接口支持多源聯邦查詢避免數據孤島產生Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Sum

63、mit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023獲取更多社區動態與最佳實踐Doris Summit 峰會官網：doris- Doris S

64、ummit 峰會回放：https:/ Doris 官網：doris.apache.orgApache Doris GitHub： Doris 官方平臺：Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023