阿里巴巴實時計算平臺_JStorm_Turbo（24頁）.pdf

《阿里巴巴實時計算平臺_JStorm_Turbo（24頁）.pdf》由會員分享，可在線閱讀，更多相關《阿里巴巴實時計算平臺_JStorm_Turbo（24頁）.pdf（24頁珍藏版）》請在三個皮匠報告上搜索。

1、TECHNOLOGIES CONSTRUCTING ELECTRONIC WORLD TRADE PLATFORM 署名：封仲淹(Longda)職稱：資深專家 Large-Scale Stream Processing inside AlibabaLarge-Scale Stream Processing inside Alibaba TECHNOLOGIES CONSTRUCTING ELECTRONIC WORLD TRADE PLATFORM 自我介紹 自我介紹 JStorm/Storm evangelist in China Our duties:Application Develop

2、ment JStorm Platform Evolving JStorm/Storm Support Cluster Management Community Commitment TECHNOLOGIES CONSTRUCTING ELECTRONIC WORLD TRADE PLATFORM 目錄 目錄 User Scenario Architecture Enhancements Performance Stability Features TECHNOLOGIES CONSTRUCTING ELECTRONIC WORLD TRADE PLATFORM JStorm Milestone

3、 JStorm Milestone 2013/4/30 0.7.1 Java core Scheduler 1.0 2013/12/30 0.9.0 Nimbus HA 1.0 Backpressure 1.0 Scheduler 2.0 2015/10/27 0.9.8.1 Koala 2.0 Scheduler 4.0 Backpressure 2.0 Topology Master 2015/11/12 2.1.0 Donate to apache Support Storm 0.10 binary Metrics Redesign Koala 3.0 Release 25 Jstorm

4、 version from 2013/4/30 0.7.1/0.9.0/0.9.1/0.9.2/0.9.3/0.9.3.1/0.9.4/0.9.4.1/0.9.5/0.9.5.1/0.9.6/0.9.6.1/0.9.6.2/0.9.6.3/0.9.7/0.9.7.1/0.9.7.2/0.9.8/0.9.8.1/0.9.8.2/2.0.4/2.1.0/2.1.1/2.2.0/2.2.12014/10/11 0.9.6.3 Koala 1.0 Scheduler 3.0 User Defined Log User Defined Metrics 1.0 2016/5/31 2.2.0 Improv

5、e performance Exactly Once Sliding Window Shade Dependency Backpressure 3.0 TECHNOLOGIES CONSTRUCTING ELECTRONIC WORLD TRADE PLATFORM 當前 當前 Everywhere 4000+machines Standalone JStorm-on-yarn JStorm-on-docker 1500+applications,2000+topologies 2+PB every day TECHNOLOGIES CONSTRUCTING ELECTRONIC WORLD

6、TRADE PLATFORM 應用場景 應用場景 Fraud Detection:Nut/Velocity Auditing:Alimama Ads auditing,Alimama P4P auditing,AMG repository auditing,Statistics:Eagleeye,AE pv?uv statistics,BI statistics Monitoring:Tlog,Rds-monitor,Oceanbase-monitor,Cainiao Raidar,QuaMonitor,YunOSMonitor,Data Transport:RDS-log-sync,Unif

7、y-Log Realtime Recommendation:Alipay 1315,Alipay Hyperloop,Tpp Scheduling Apps:AE mail realtime analyzer,1688 Wangxiaobao TECHNOLOGIES CONSTRUCTING ELECTRONIC WORLD TRADE PLATFORM JStorm 生態 JStorm 生態 TECHNOLOGIES CONSTRUCTING ELECTRONIC WORLD TRADE PLATFORM 架構 架構 TLog Kepler NUT ARM RDS Trip YunOS C

8、ainiao Porter(Koala)SQL engine Plugin（Kafka/Meta/TT/Mysql/HDFS/Hbase/）CEP Plugin Trident Exactly Once Extension API Raw Storm API JStorm Core Local Cluster Standalone,YARN,Fuxi Cloud(Docker,ECS,.)TECHNOLOGIES CONSTRUCTING ELECTRONIC WORLD TRADE PLATFORM 0 2 4 6 8 10 12 2 worker,1 spout,2 split,2 cou

9、nt 10 worker,5 spout,20 split,20 count 10 worker,10 spout,40 split,40 count 20 worker,10 spout,80 split,80 count 40 worker,10 spout,160 split,160 count 80 worker，10 spout，320 split，320 count Million JStorm Storm 性能 性能 HW:32 CPU core/128G Storm version:1.0.2 Test:Storm FastWordCount JStorm version:2.

10、2.0 Worker:6G RAMTECHNOLOGIES CONSTRUCTING ELECTRONIC WORLD TRADE PLATFORM Pipeline Application Deploy Core 性能 性能 Task Pipeline Tuning UserDefined scheduler Improve efficiency Improve parallel Shuffle Resolve bottleneck Batch DesignSerialization TuningSeparate spout send/recvMetrics TuningRouting Op

11、timizationOverhead ReductionTECHNOLOGIES CONSTRUCTING ELECTRONIC WORLD TRADE PLATFORM Isolation High Availability Monitoring Degradation Multi-Cluster Deployment 穩定性 穩定性 Backpressure UserDefined degrade Message lifecycle Userdefined metrics Userdefined alarm Single failure:nimbusHA Node self check:c

12、heckhealth Huge deploy:topology-master Across cluster:jstorm-on-docker/standalone In one cluster:jstorm-on-yarn/cgroup Clusters in different city:hot backup/cold backup Clusters in one City TECHNOLOGIES CONSTRUCTING ELECTRONIC WORLD TRADE PLATFORM 監控系統 監控系統 TECHNOLOGIES CONSTRUCTING ELECTRONIC WORLD

13、 TRADE PLATFORM 自定義監控 自定義監控 Improving performance:Multiple Layer Metrics Design Automatically aggregate task levels metric to component level,then aggregate into topology levelShare common metricsTECHNOLOGIES CONSTRUCTING ELECTRONIC WORLD TRADE PLATFORM 自定義監控 自定義監控 TECHNOLOGIES CONSTRUCTING ELECTRON

14、IC WORLD TRADE PLATFORM NimbusTopology MasterTaskZK ClusterTopologyTaskTaskTopology MasterTaskTaskTaskTopology MasterTaskTopologyTaskTaskTopology MasterMetricsTask heartbeatControl event大規模部署 大規模部署 TopologyTopology Master slaveTECHNOLOGIES CONSTRUCTING ELECTRONIC WORLD TRADE PLATFORM JStorm Backpres

15、sure 3.0反壓 反壓 Netty clientNetty cacheSending taskNetty serverTask cacheReceive taskIn one workerTask cacheReceive taskExceed ThresholdTECHNOLOGIES CONSTRUCTING ELECTRONIC WORLD TRADE PLATFORM 其他強化功能 其他強化功能 User Side Functionalities User-Defined Scheduler User-Defined Resource Usage Advanced Logging

16、Mechanism More Fine-Grained Metrics Graceful Shutdown Dynamic Rescale/Reload/Restart Receive Command Quick Application Upgrade TECHNOLOGIES CONSTRUCTING ELECTRONIC WORLD TRADE PLATFORM Exactly OnceExactly Once Chandy-Lamport Algorithm Barrier Stream Aligning TECHNOLOGIES CONSTRUCTING ELECTRONIC WORL

17、D TRADE PLATFORM Spout Stateless bolt State bolt Ending bolt Topology Master Cache dbHdfs/hbaseNormal batchControllerNexttupleCommitCache dbExecuteExecuteCommitExecuteExecuteCommit offsetCheckpoint barrierCheckpoint barrierJStorm checkpoint(optional)Jstorm checkpointCheckpoint barrierCheckpoint barr

18、ierCheckpoint and clear old checkpointExactly OnceExactly Once TECHNOLOGIES CONSTRUCTING ELECTRONIC WORLD TRADE PLATFORM Exactly OnceExactly Once Spout Stateless bolt State bolt Ending bolt Topology Master Cache dbHdfs/hbaseControllerNexttupleRollbackCache dbExecuteExecuteRollbackExecuteExecuteRollb

19、ack barrierRollback barrierRollback barrierRollback barrierRead last success checkpoint and clean useless checkpoint startrollbackRead checkpoint(optional)rollbackTECHNOLOGIES CONSTRUCTING ELECTRONIC WORLD TRADE PLATFORM FutureFuture Apache Beam A more powerful high level programming framework Easie

20、r to learn,to debug Provide higher throughput TECHNOLOGIES CONSTRUCTING ELECTRONIC WORLD TRADE PLATFORM 開源現狀 開源現狀 50+CompaniesTECHNOLOGIES CONSTRUCTING ELECTRONIC WORLD TRADE PLATFORM 總結 總結 Why Start JStorm?Move faster Whats the difference between JStorm and Storm?JStorm means Stream Process Ecosystem JStorm is Enterprise Java Storm More faster More stable More featureTECHNOLOGIES CONSTRUCTING ELECTRONIC WORLD TRADE PLATFORM zhongyan.fengalibaba-?www.jstorm.io