5.觀測云-基于Apache Doris 構建新一代的可觀測平臺.pdf

《5.觀測云-基于Apache Doris 構建新一代的可觀測平臺.pdf》由會員分享，可在線閱讀，更多相關《5.觀測云-基于Apache Doris 構建新一代的可觀測平臺.pdf（18頁珍藏版）》請在三個皮匠報告上搜索。

1、觀測云-基于Apache Doris 構建新一代的可觀測平臺蔣爍淼觀測云-CEODoris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia

2、2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 20231什么是觀測云Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Dor

3、is Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023觀測云的原理Doris Summit Asia 2023Doris Summit Asia 2023Doris Summi

4、t Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Dor

5、is Summit Asia 2023Doris Summit Asia 2023觀測云的能力Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summi

6、t Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023觀測云的架構Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2

7、023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023GuanceDB 原有架構Storage EngineElasticSearch/OpenSearchMetr

8、ic StorepipelineQuery EngineparserplanneroptimizerexecutordatasourceDoris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit

9、 Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023觀測云的以往痛點 寫入占用集群性能過大 面對用戶的自定義數據對 SchemaLess 支持有限 ElasticSearch（Opensearch）的面對超海量數據的聚合查詢能力有限 多租戶情況下的隔離效果差 高級分析能力增強有限Doris Summit Asia 2023

10、Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit A

11、sia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023其他方案為什么不可行 SaaS 需要支持一定的查詢高并發 日志場景，倒排索引是必須的 SchemaFix 并不適合可觀測性場景 多租戶情況下，大量的元數據是否造成性能問題 是否能支持面向可觀測性場景的數據查詢能力Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summ

12、it Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023GuanceDB 整體架構Metric Stor

13、epipelineQuery EngineparserplanneroptimizerexecutordatasourceStorage EngineDoris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris

14、 Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023GuanceDB for DorisDoris ClusterDoris FE Doris BE Guance-SelectSQL&MySQL DriverBacth Stream LoadInternal RPCDQLGuanceDBGuance-Inse

15、rtstructure-dataArrow Format&Thrift RPCDoris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2

16、023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023DQL vs SQLDQL查詢 專為可觀測場景設計，表達簡潔，功能強大 支持所有查詢負載，Metrics、Logs、Metrics、Objects.等 All In OneDQL 優勢 M:cpu:(avg(cpu_usage)cluster=testing 1h:5m:avg M:httpserver:(derivative(

17、request_count)by account_id sorder by account_id desc slimit 10 L:(L:*limit 1000):(count(_docid)as count,collect(_docid,50)by drain(message,0.3)as sample sorder by count desc語意相似的 SQL 模擬 step+rollup：select ts,avg(cpu_usage)from(select cast(floor(millisecond_timestamp(time)/300)*300 AS bigint)as ts,a

18、vg(cpu_usage)as cpu_usage from M where time=now()-1h and time now()and cluster=test group by ts,instance)group by ts sorder by 無法實現，缺乏 derivative 等時序算子，drain 日志聚類無法實現Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Do

19、ris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Variant 設計業務場景數據類型多變、降低運維負擔可觀測場景為什么需要

20、Variant?Variant 的優勢：每個 Partition 動態創建，數據變化后字段可以失效 沒有類型沖突，支持多類型合并升級動態 schema 為什么不行？查詢時根據表 Schema 判斷是否是 Variant 字段，以及根據用戶 DQL 中的意圖推斷期望的數據類型 完全用戶透明，DQL 和寫入都無需改動，我們自動處理 VariantGuanceDB 中怎么適配？我們在 ES 中遇到的問題：類型沖突 字段數量上限Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023

21、Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit A

22、sia 2023聚合采樣 原始數據量級大，查詢開銷極高 實際業務確實不需要精確計數為什么需要聚合采樣?GuanceDB 中怎么做采樣？查詢性能提升 20 倍 業務透明，自動根據數據量進行聚合采樣，特殊需要可以關閉采樣最終的效果：評估查詢起始范圍內原始數據行數 當原始行數大于 100w 時開啟采樣，固定采樣行數為 100w 來計算采樣率 利用 doris tablesample 能力進行采樣 根據不同的算子自動對結果按照比例進行伸縮，主要是 sum+countDoris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Dor

23、is Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia

24、 2023Doris Summit Asia 2023性能、成本估算Up to4x快速查詢Up to2x綜合成本下降67%查詢性能普通查詢ElasticsearchDorisElasticsearch20臺主機Doris13臺主機Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Sum

25、mit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023副產品-Doris 的完整監控觀測方案Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Dori

26、s Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia

27、2023Doris Summit Asia 2023可能是目前最好的監控觀測方案 將實時數倉的能力應用到了監控日志及更廣泛的可觀測性場景 結合觀測云的產品力，無需再針對 Doris 進行二次開發即可擁有完整的監控觀測能力 Doris帶來的強大實時分析能力將利用Metrics，Logs，Traces等可觀測性數據的價值提升了一個維度。整體性價比傳統的開源方案提升至少 20 倍的性價比，同時帶來更好的用戶體驗 開放 Doris 的數倉能力還可以將可觀測性數據與業務數據通過 SQL 進行整合，讓可觀測性數據發揮更大價值 可以更有能力構建更多面向業務場景的實時觀測監控場景好用不貴，極大的提升整個產研團

28、隊的效率，提升整個產研團隊能力邊界Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2

29、023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023歡迎大家注冊使用體驗Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023D

30、oris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023目錄獲取更多社區動態與最佳實踐Doris Summit 峰會官網：doris- Doris Summit 峰會回放：https:/ Doris 官網：doris.

31、apache.orgApache Doris GitHub： Doris 官方平臺：Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023