a-story-of-managing-kubernetes-watch-events-end-to-end-flow-in-extremely-large-clusters-nanomao-zhong-shi-fa-lia-kubernetes-watchguo-guo-xiao-zha-bo-tang-ant-group.pdf

《a-story-of-managing-kubernetes-watch-events-end-to-end-flow-in-extremely-large-clusters-nanomao-zhong-shi-fa-lia-kubernetes-watchguo-guo-xiao-zha-bo-tang-ant-group.pdf》由會員分享，可在線閱讀，更多相關《a-story-of-managing-kubernetes-watch-events-end-to-end-flow-in-extremely-large-clusters-nanomao-zhong-shi-fa-lia-kubernetes-watchguo-guo-xiao-zha-bo-tang-ant-group.pdf（26頁珍藏版）》請在三個皮匠報告上搜索。

1、A Story of Managing K8s Watch Events End-to-end Flow in Extremely Large ClustersBo Tang,Ant GroupAug,2024OutlineThe Watch Mechanism and The Importance of Watch in KubernetesDefinition of Watch SLO and What It Brings to UsWhat We Did to Optimize Watch Flow and the Benefits We GotSummary and Future Pl

2、ansWatchMechanismWatchSLOOptimizationSummaryK8s OverviewKubernetes is getting boring?Let us investigate K8s moreYou will get what you wantStable SkeletonComprehensive extensibilityDesign PhilosophyAfter KubernetesBefore KubernetesK8s Birds-eyes ViewSchedulerKubeletControllersWhen Cluster Gets LargeK

3、CMSchedulerKubeletCustom ControllersKube-proxyWhen Cluster Gets LargeProblems arise when cluster gets large:Large number of nodes and pods.Large number of CRDsHeavy traffichigh churn rate of resources.The provisioning path is longDifferent usages of different controllersUsers requirements diverse/va

4、ry a lotAlmost every problem is connected to Watch mechanism.Weve done optimization for each component of Kubernetes system,but the link/connection part seems missing.Watch Procedure1.Apiserver obtains data from etcd,decodes it,formats it,and sends it to apiservers internal cache.2.Based on the watc

5、hed data,apiserver builds its cache internally,which is called the WatchCache object.It includes a full set of data and a circular array.*The full set of data is a set of data that includes indexing and is used for various client list requests,list-by-label requests,and list-by-field requests.*The c

6、ircular array contains the latest watchCache event and is used for various client watch requests.3.The apiserver internally iterates through all of the cacheWatchers for a resource and sends each cacheWatcher the change events it is interested in one by one.4.When the client receives data from apise

7、rver,it performs decoding,and puts it into the clients cache.At the same time,generate corresponding add/update/delete events and put them into the workqueue.5.The clients user code reads the workqueue and performs corresponding reconcile reconciliation processing.WatchMechanismWatchSLOOptimizationS

8、ummaryK8s SLO Definitionhttps:/ SLOSLI：The SLI is defined as the duration from the time an event enters the APIserver until it leaves the APIserver,effectively measuring as the ApiserverTimeSpan.Measured as P99 over last 1 min.SLO：The SLO is defined as the latency of all events from entering the API

9、 server until they exit the API server.Within a rolling window of 1min,the 99th percentile of this latency time should be less than 1 second,which is mathematically represented as P99(ApiserverTimeSpan)14.05s，a 59.2%enchancementAnd the apiserver network bandwidth decreases from 20GB to an average of

10、 7.5 GBWatchMechanismWatchSLOOptimizationSummarySummary2x to 10-90 x times reduction in terms of the following metrics:P99/P999 RT from etcd to apiserver watchcacheP99/P999 RT from apiserver to client sideClient side relist/rewatch number reduction and the delay caused by the full relistThe end-to-e

11、nd pod provisioning time,from pod creation to running.Apiserver load balancer bandwidth reductionThe scalability/stability of the whole systemPositively Affects billions of watch events per day.Positive feedback from internal users(Alipay ads/search,risk management,hpa,etc)and supports Alibaba/Alipa

12、y 11.11 for recent years BandwidthRelistRewatchRTPod ProvisionAdsSearchRisk11.11You will get what you wantvery soonSummary and Road AheadAre we done yet?NoThere are still problems going onThe reduction of client side CPUThe reduction of lock contention for arbitrary client codeThe proper adjustment of apiserver watch cacheThe proper usage of client for removing 409The etcd watch guaranteeTraffic measurement and analysisThe inclusion of higher version K8s featureThank youYou will get what you wantvery soonReferencehttps:/ 5ImageSrc:https:/