ACI 轉發網絡工程師藍圖.pdf

《ACI 轉發網絡工程師藍圖.pdf》由會員分享，可在線閱讀，更多相關《ACI 轉發網絡工程師藍圖.pdf（144頁珍藏版）》請在三個皮匠報告上搜索。

1、#CiscoLive#CiscoLiveJoe Young,ACI Technical Leader,Customer ExperienceBRKDCN-3900A Network Engineers Blueprint for ACI Forwarding 2023 Cisco and/or its affiliates.All rights reserved.Cisco Public#CiscoLiveEnter your personal notes hereCisco Webex App Questions?Use Cisco Webex App to chat with the sp

2、eaker after the sessionFind this session in the Cisco Live Mobile AppClick“Join the Discussion”Install the Webex App or go directly to the Webex spaceEnter messages/questions in the Webex spaceHowWebex spaces will be moderated by the speaker until June 9,2023.12343https:/ 2023 Cisco and/or its affil

3、iates.All rights reserved.Cisco PublicBRKDCN-39003Agenda 2023 Cisco and/or its affiliates.All rights reserved.Cisco PublicWhats Different About ACI Forwarding?(iVXLAN,contracts,endpoint learning)Proxy ForwardingACI Forwarding TablesEndpoint tables,routing tables,hardware lookupsUnderstanding the Con

4、figuration OptionsThe Anatomy of an ACI SwitchBRKDCN-39004Agenda 2023 Cisco and/or its affiliates.All rights reserved.Cisco PublicUnderstanding the ToolsUI ToolsElamFtriageSpan/ERSPANFlow Telemetry/netflowDebugging and Walking Through ACI Flows(Routed,Bridged,BUM,Proxied)BRKDCN-39005 2023 Cisco and/

5、or its affiliates.All rights reserved.Cisco Public#CiscoLiveGlossary of AcronymnsAcronymsAcronymsDefinitionsDefinitionsACIApplication Centric InfrastructureAPICApplication Policy Infrastructure ControllerEPEndpointEPGEndpoint GroupBDBridge DomainVRFVirtual Routing and ForwardingCOOPCouncil of Oracle

6、 ProtocolVxLANVirtual eXtensible LANAcronymsAcronymsDefinitionsDefinitionsdXXXoOuter Destination XXX(dIPo=Outer Destination IP)sXXXoOuter Source XXX(sIPo=Outer Source IP)dXXXiInner Destination XXX(dIPi=Inner Destination IP)sXXXiInner Source XXX(sIPi=Inner Source IP)GIPoOuter Multicast Group IPVNIDVi

7、rtual Network IdentifierVxLAN packet acronymsBRKDCN-39006Whats Different About ACI Forwarding?2023 Cisco and/or its affiliates.All rights reserved.Cisco Public#CiscoLiveWhat is“Application Centric”?Traditional networks use ACLs to classify trafficUsually based on L3 or L2 addresses Makes security de

8、cisions(permit,deny,log,etc)Makes forwarding decisions(policy based routing)ACI can classify traffic based on its EPGTraffic inherits the forwarding and security policy of the EPGHost1Host1EPG1EPG1Host2Host2EPG2EPG2Host3Host3EPG3EPG3AppAppEPG4EPG4BRKDCN-39008 2023 Cisco and/or its affiliates.All rig

9、hts reserved.Cisco Public#CiscoLiveHow is“Application Centric”Achieved?Sources and Destinations Must be Classified into EPGsEndpointsPolicy-PrefixesUsed by App EPGsRepresents the network identity of an end deviceLearned dynamically or configured staticallyUsed by External EPGsClassifies destination

10、by longest prefix matchAlso used for shared-servicesConfiguredPcTagsThe security ID of an EPGUsed in contracts.Ex:Permit PcTag 1000 to PcTag 2000Sclass/dclass imply PcTag directionContractsDefines security and sometimes forwarding(pbr)policy between epgsEssentially an ACL between PcTagsConsumer/Prov

11、ider rather than src/destBRKDCN-39009 2023 Cisco and/or its affiliates.All rights reserved.Cisco Public#CiscoLiveVlan TypesInternal ID on LEAF(not shared across LEAFs)For forwarding(global value for entire fabric)VLAN ID for external devices(user configured value)VRF1BD1EPG1vlanvlan-5 51919171712661

12、12661166132591661325925231362523136vxlanvxlan-838860883886082020Access Access EncapEncap VLANVLANPIPI-VLANVLANVxLANVxLAN IDID(VNID)(VNID)83886088388608EPEPvlanvlan-5 530303131Access Access EncapEncap VLANVLANPIPI-VLANVLANEPLEAF 1LEAF 1EPVRF1BD1EPG1LEAF 2LEAF 2vxlanvxlan-838860883886083333For BD SVIF

13、or BD SVI PIPI-VLAN:Platform Independent VLANVLAN:Platform Independent VLANBRKDCN-390010 2023 Cisco and/or its affiliates.All rights reserved.Cisco Public#CiscoLiveWhat is an Endpoint?An Endpoint joins both forwarding and security policyleaf103#show system internal epm end ip 192.168.200.11show syst

14、em internal epm end ip 192.168.200.11MAC:0000.1111.2222:Num IPs:1IP#0:192.168.200.11:IP#0 flags:l3-sw-hit:NoVlan id:2:Vlan vnid:1266112661:VRF name:CL2022:vrf1BD vnid:1661325916613259:VRF vnid:25231362523136Phy If:0 x40018000:Tunnel If:0Interface:Ethernet1/25/1Ethernet1/25/1Flags:0 x80005c04:sclass:

15、32771 sclass:32771:Ref count:5EP Create Timestamp:11/01/2021 14:06:25.769904EP Update Timestamp:11/04/2021 18:51:54.387104EP Flags:local|IP|MAC|host-tracked|sclass|timer|leaf103#show system internal epm endpoint ip 192.168.100.10show system internal epm endpoint ip 192.168.100.10MAC:0000.0000.0000:N

16、um IPs:1IP#0:192.168.100.10:IP#0 flags:l3-sw-hit:NoVlan id:0:Vlan vnid:0:VRF name:CL2022:vrf1BD vnid:0:VRF vnid:25231362523136Phy If:0:Tunnel If:0 x18010001Interface:Tunnel1Tunnel1Flags:0 x80004400:sclass:49154 sclass:49154:Ref count:3EP Create Timestamp:11/04/2021 16:38:13.570615EP Update Timestamp

17、:11/04/2021 18:51:54.386595EP Flags:IP|sclass|timer|Local LearnRemote LearnPcTagPcTagInterface/TEPInterface/TEPVNIDVNIDBRKDCN-390011 2023 Cisco and/or its affiliates.All rights reserved.Cisco Public#CiscoLiveWhat is a TEP?(Tunnel Endpoint)IP addresses allocated for overlay communicationVXLAN Traffic

18、 is sent to the TEP+VNID of destinationTEP TypeWhat is it?What is it for?Physical TEP(PTEP)Unique Overlay IP Address for each individual Leaf/SpineNon-vpc dataplane,l3out communication,apic-leaf comm,etcVPC TEP(VTEP)Unique Overlay IP Address for each VPC PairTraffic destined to endpoints that are co

19、nnected behind VPCProxy TEPSpine Anycast IPs used for proxy trafficLeafs send to these TEPs when doing proxy forwardingMost Common TEP TypesMost Common TEP Typesa-leaf101#show ip interface loopback0IP Interface Status for VRF overlay-1lo0,Interface status:protocol-up/link-up/admin-up,iod:4,mode:ptep

20、BRKDCN-390012 2023 Cisco and/or its affiliates.All rights reserved.Cisco Public#CiscoLiveWhat are Tunnels?Leafs/Spines Install Tunnel Interface to each known TEP.Used for VXLAN Dataplaneleaf#moquery-c tunnelIf-f tunnel.If.id=tunnel1id :tunnel1dest:10.0.72.67idRequestorDn:sys/*/db-dtep/dtep-10.0.72.6

21、7leaf#moquery-c tunnelIf-f tunnel.If.id=tunnel1id :tunnel1dest:10.0.72.64idRequestorDn:sys/bgp/*/db-dtep/dtep-10.0.72.64leaf#moquery-c tunnelIf-f tunnel.If.id=tunnel1#tunnel.Ifid :tunnel1dest:10.0.152.64idRequestorDn:sys/isis/*/lvl-l1/db-dtep/dtep-10.0.152.64Dataplane LearnsThrough BGP(l3out routes)

22、Local POD ISIS DatabaseHow are Tunnels Learned?How are Tunnels Learned?BRKDCN-390013 2023 Cisco and/or its affiliates.All rights reserved.Cisco Public#CiscoLiveHow is an Endpoint Learned?LeafLeafLeafLeafSpineSpineSpineSpineEP2EP210.1.1.2/2410.1.1.2/244444.5555.66664444.5555.6666EP1EP110.1.1.1/2410.1

23、.1.1/240000.1111.22220000.1111.2222Ingress leaf classifies smac and sIP Ingress leaf classifies smac and sIP(if IP learning enabled)into EPG(if IP learning enabled)into EPG based on some info such as vlan.based on some info such as vlan.Endpoint entry installedEndpoint entry installedLeafLeafLeaf Up

24、dates COOP Leaf Updates COOP Database on spinesDatabase on spinesSource sends some Source sends some type of traffictype of trafficLeaf Installs Remote Leaf Installs Remote Endpoint learn from Endpoint learn from dataplanedataplaneHow does the Egress leaf classify traffic into the correct EPG?4312BR

25、KDCN-390014 2023 Cisco and/or its affiliates.All rights reserved.Cisco Public#CiscoLiveOverlay iVXLAN ACI uses VXLAN with some additional bitsLeafLeafLeafLeafSpineSpineSpineSpineEP2EP210.1.1.2/2410.1.1.2/244444.5555.66664444.5555.6666EP1EP110.1.1.1/2410.1.1.1/240000.1111.22220000.1111.2222LeafLeafSI

26、PDIPDSCPDMACSMAC802.1QVXLANSIPDIPProtoL4/PayloadDMACSMAC802.1QPcTag/Sclass(2 bytes)VNID(3 bytes)Bit pos 4 Source Policy AppliedBit pos 5 Destination Policy AppliedBit pos 7 Dont learnFlags(1 byte)Dataplane VXLAN contains all information needed for endpoint classificationBRKDCN-390015 2023 Cisco and/

27、or its affiliates.All rights reserved.Cisco Public#CiscoLiveHow is an Endpoint Learned?EP1EP110.1.1.1/2410.1.1.1/240000.1111.22220000.1111.2222LeafLeafEncap Vlan 100leaf103#show vlan encapshow vlan encap-id 100id 100VLAN Name Status Ports-2 CL2022:ap1:epg2CL2022:ap1:epg2active Eth1/25/3leaf103#show

28、system internal epm vlan 2 detailshow system internal epm vlan 2 detailVLAN 2VLAN 2VLAN type:FD vlanhw id:34:sclass:3277132771access enc:(802.1Q,100100)fabric enc:(VXLAN,1266112661)Object store EP db version:4BD vlan id:1:BD vnid:1661325916613259:VRF vnid:25231362523136Valid:Yes:Incomplete:No :Learn

29、 Enable:YesPIPI-VLANVLANSpineSpineEP Sends EP Sends some trafficsome trafficBRKDCN-390016 2023 Cisco and/or its affiliates.All rights reserved.Cisco Public#CiscoLiveChecking EndpointsReference commands can be run from leafs or apics#Check object model for Mac Address Endpoint#Check object model for

30、Mac Address Endpointmoquery-c epmMacEp-f epm.MacEp.addr=00:00:AA:AA:BB:BB#Check object model for IP Address Endpoint#Check object model for IP Address Endpointmoquery-c epmIpEp-f epm.IpEp.addr=192.168.200.11Reference commands can be run from leafs only#Check endpoint manager process directly#Check e

31、ndpoint manager process directlyshow system internal epm endpoint mac 0000.aaaa.bbbbshow system internal epm endpoint ip 192.168.200.11#Check hardware level endpoint process directly#Check hardware level endpoint process directlyvsh_lc-c show system internal epmc endpoint mac 0000.aaaa.bbbbvsh_lc-c

32、show system internal epmc endpoint ip 192.168.200.11BRKDCN-390017 2023 Cisco and/or its affiliates.All rights reserved.Cisco Public#CiscoLiveHow is an Endpoint Learned?EP1EP110.1.1.1/2410.1.1.1/240000.1111.22220000.1111.2222LeafLeafspine1005#show coop internal info ipshow coop internal info ip-db|gr

33、ep db|grep-B 1 B 1-A 15 A 15 192.168.200.11192.168.200.11-IP address:192.168.200.11Vrf:25231362523136Flags:0EP bd vnid:1661325916613259EP mac:00:00:AA:AA:BB:BBPublisher Id:10.0.64.70Record timestamp:11 05 2021 17:02:56 217794556Publish timestamp:11 05 2021 17:02:56 220584642Seq No:0Remote publish ti

34、mestamp:01 01 1970 00:00:00 0URIB Tunnel InfoNum tunnels:1Tunnel address:10.0.64.7010.0.64.70Tunnel ref count:1SpineSpineUpdate Update COOPCOOPThe Leaf Updates COOP on SpinesVNID info should match the info on leafLeaf TEP that owns this EP:#From APICmoquery moquery-c ipv4Addr c ipv4Addr-f ipv4.Addr.

35、addr=10.0.64.70f ipv4.Addr.addr=10.0.64.70BRKDCN-390018 2023 Cisco and/or its affiliates.All rights reserved.Cisco Public#CiscoLiveChecking COOPReference commands can be run from spines or apicsQuery COOP for l2 entry:Query COOP for l2 entry:moquery-c coopEpRec-f coop.EpRec.mac=00:00:AA:AA:BB:BBQuer

36、y COOP for l3 entry and get parent l2 entry:Query COOP for l3 entry and get parent l2 entry:moquery-c coopEpRec-x rsp-subtree=children rsp-subtree-filter=eq(coopIpv4Rec.addr,1.1.1.1)rsp-subtree-include=requiredQuery COOP for l3 only entry(such as an SVI IP):Query COOP for l3 only entry(such as an SV

37、I IP):moquery-c coopIpOnlyRec-f coop.IpOnlyRec.addr=192.168.100.10Query COOP for l3 ep:Query COOP for l3 ep:moquery-c coopIpv4Rec-f coop.Ipv4Rec.addr=192.168.100.10BRKDCN-390019 2023 Cisco and/or its affiliates.All rights reserved.Cisco Public#CiscoLiveHow is Traffic Classified with no EP Learn?In m

38、ost of these cases,the pcTag is based on a policy-prefix lookupThere will be no endpoint learn in several casesSource/dest is behind an l3outSource/dest is in another vrfEndpoint learning is disabled by some optionIf ingress leaf doesnt apply policy,egress leaf should(indicated via policy-applied bi

39、ts in ivxlan header)BRKDCN-390020 2023 Cisco and/or its affiliates.All rights reserved.Cisco Public#CiscoLiveHow is Traffic Classified with no EP Learn?Destination Behind L3outleaf101#vsh_lc vsh_lc-c show forwarding route 10.99.99.100 platform vrf CL2022:vrf1c show forwarding route 10.99.99.100 plat

40、form vrf CL2022:vrf1!Policy Prefix 10.99.99.0/24Policy Prefix 10.99.99.0/24!vrf:16(0 x10),routed_if:0 x0 epc_class:3277232772(0 x8004)Classification based on longest l3out policy prefixBRKDCN-390021 2023 Cisco and/or its affiliates.All rights reserved.Cisco Public#CiscoLiveHow is Traffic Classified

41、with no EP Learn?Destination is unknown and is proxiedleaf101#show show ipip route 192.168.200.20 route 192.168.200.20 vrfvrf CL2022:vrf1CL2022:vrf1192.168.200.0/24,ubest/mbest:1/0,attached,direct,pervasivepervasive*via 10.0.176.66%overlay-1,1/0,4d05h,static,tag 4294967294recursive next hop:10.0.176

42、.66/32%overlay-1leaf101#vsh_lcvsh_lc-c show forwarding route 192.168.200.20 platform c show forwarding route 192.168.200.20 platform vrfvrf CL2022:vrf1CL2022:vrf1!Policy Prefix 0.0.0.0/0!Vrf:16(0 x10),routed_if:0 x0 epc_classepc_class:1(0 x1):1(0 x1)-pcTag of 1 indicates the fabric owns the subnet,d

43、ont apply policy-policy applied flags not set in ivxlan header“Pervasive”indicates this is a BD or EPG subnet(fvSubnet).Send to spine proxy-addrleaf101#show isis dtep vrf overlayshow isis dtep vrf overlay-1|egrep Type|PROXY1|egrep Type|PROXYDTEP-Address Role Encapsulation Type10.0.176.66 SPINE N/A P

44、HYSICAL,PROXYPROXY-ACASTACAST-V4V410.0.176.65 SPINE N/A PHYSICAL,PROXYPROXY-ACASTACAST-MACMAC10.0.176.64 SPINE N/A PHYSICAL,PROXYPROXY-ACASTACAST-V6V6Dont apply policy,Forward to proxy Anycast!BRKDCN-390022 2023 Cisco and/or its affiliates.All rights reserved.Cisco Public#CiscoLiveleaf#show show ipi

45、p route 192.168.255.10 route 192.168.255.10 vrfvrf CL2022:vrf1CL2022:vrf1192.168.255.0/24,ubest/mbest:1/0,attached,direct,pervasivepervasive*via 10.0.176.66%overlay-1,1/0,static,tag!,rwVnidrwVnid:vxlan:vxlan-24576012457601recursive next hop:10.0.176.66/32%overlay-1leaf#vsh_lcvsh_lc-c show forwarding

46、 route 192.168.255.10 plat c show forwarding route 192.168.255.10 plat vrfvrf CL2022:vrf1CL2022:vrf1Prefix:192.168.255.0/24,Update_time:Fri Nov 5 20:57:00 2021!Policy Prefix 0.0.0.0/0!Flags:IN-HW,SHRDSHRD-SVCSVC,vrf:16(0 x10),routed_if:0 x0 epc_classepc_class:36(0 x24):36(0 x24)leaf#show show ipip r

47、oute 192.168.100.10 route 192.168.100.10 vrfvrf CL2022:vrf2CL2022:vrf2192.168.100.0/24,ubest/mbest:1/0,attached,direct,pervasivepervasive*via 10.0.176.66%overlay-1,1/0,static,rwVnidrwVnid:vxlan:vxlan-25231362523136recursive next hop:10.0.176.66/32%overlay-1leaf#vsh_lcvsh_lc-c show forwarding route 1

48、92.168.100.10 plat c show forwarding route 192.168.100.10 plat vrfvrf CL2022:vrf2CL2022:vrf2Prefix:192.168.100.0/24,Update_time:Tue Nov 9 14:34:05 2021!Policy Prefix 0.0.0.0/0!Flags:IN-HW,SHRDSHRD-SVC,SVC,vrf:10(0 xa),routed_if:0 x0 epc_classepc_class:14(0 xe):14(0 xe)How is Traffic Classified with

49、no EP Learn?Destination is in shared services providerprovider EPG(different vrf)Destination is in shared services consumerconsumer EPG(different vrf)PcTag of provider epgPcTag of provider epgReserved tag for shared Reserved tag for shared services consumer.Policy services consumer.Policy applied in

50、 consumer vrfapplied in consumer vrfShared Services ClassificationBRKDCN-390023 2023 Cisco and/or its affiliates.All rights reserved.Cisco Public#CiscoLiveleaf#show zoning-rule scope 2523136 src-epg 200+-+-+-+-+-+|Rule ID|SrcEPG|DstEPG|FilterID|Action|+-+-+-+-+-+|4159|200|100|532|permit|+-+-+-+-+-+C

51、ontracts and ForwardingIngressDestDestEPG1EPG1PcTag 100 PcTag 100 SourceSourceEPG2EPG2PcTag 200PcTag 200LeafLeafLeafLeafContract Found?Set policy-applied bits in ivxlan.Permit,deny,redir,logIf LPM is BD/EPG subnet,forward and dont set policy-applied bits in ivxlan.Otherwise,drop!YesNoPolicy-Applied

52、Bits set?Dont do contract lookup.Forward.Do contract lookup.Permit,deny,redir,logYesNoEgressCheck hidden slide for impact of“Policy Control Enforcement Direction”settingBRKDCN-390024 2023 Cisco and/or its affiliates.All rights reserved.Cisco Public#CiscoLivePolicy enforcement tableWhere is policy en

53、forced?Flow DirectionFlow DirectionINGRESSINGRESSEGRESSEGRESSEPG to unknown EPGApplied EgressUnchangedEPG to known EPGApplied IngressUnchangedEPG to L3outApplied Ingress/non-BLApplied Egress/BLL3out to unknown EPGApplied Egress/non-BLApplied EgressL3out to known EPGApplied Egress/non-BLApplied Ingre

54、ss/BLL3out to L3outApplied IngressApplied EgressPolicy enforcement affects only traffic to or from the L3Out.There are no behavior changes in EPG-to-EPG.VRF Enforcement SettingBRKDCN-390025 2023 Cisco and/or its affiliates.All rights reserved.Cisco Public#CiscoLiveWhat About Flooded Traffic?The foll

55、owing traffic may be flooded:Flooded traffic is sent to the BD GiPo(l2 flood)or VRF GiPo(l3 flood)The GiPo is an overlay multicast address allocated to a BD or VRFFlooding is done on a loop-free tree called an FTAGSecurity policy NOT appliedBroadcastMulticastUnknown UnicastControl Plane maintenance(

56、EP announce,fabric ARP,etc)How does ACI flood?GiPoBRKDCN-390026 2023 Cisco and/or its affiliates.All rights reserved.Cisco Public#CiscoLiveWhat are FTAGs?FTAGs are loop-free trees within the overlay used by flooded trafficFTAGs are picked per flow from values 0 0 xcOne spine is root for each treeOut

57、going interfaces calculated by ISISLeafLeafLeafLeafSpineSpineSpineSpineEP1EP110.1.1.1/2410.1.1.1/240000.1111.22220000.1111.2222LeafLeafWho has 10.1.1.100?Who has 10.1.1.100?Please tell 10.1.1.1Please tell 10.1.1.1ARPARPSelect ftag 0,forward Select ftag 0,forward out root port*out root port*Root for

58、Root for Ftag 0Ftag 0Forward out all Outgoing Forward out all Outgoing InterfacesInterfaces*Note,the ingress leaf communicates the selected ftag to the rest of the fabric by adding it to the destination gipo.If the gipo is 225.0.0.0 and the ftag is 0 x9,the destination address would be 225.0.0.9321B

59、RKDCN-390027 2023 Cisco and/or its affiliates.All rights reserved.Cisco Public#CiscoLiveChecking FTAGsFind the outgoing interfaces for a treeleaf101#show isis internal mcast routes ftagshow isis internal mcast routes ftagFTAG Routes=FTAG ID:0 Enabled Cost:(1/7/0)-Root port:Ethernet1/54.6 Root port:E

60、thernet1/54.6 OIF List:Ethernet1/53.5!ommitted rest of ftagsspine1005#show isis internal mcast routes ftagshow isis internal mcast routes ftagFTAG Routes=FTAG ID:0 Root Root Enabled Cost:(0/0/0)-Root port:-OIF List:Ethernet1/1.20Ethernet1/1.20Ethernet1/2.21Ethernet1/2.21Ethernet1/3.19Ethernet1/3.19!

61、ommitted rest of ftagsThis spine is the root for ftag 0Forward out all of these interfacesLeaf forwards to root port and any additional OIFsCheck FTAG tree on ingress leafCheck FTAG tree on root spineBRKDCN-390028Proxy Forwarding 2023 Cisco and/or its affiliates.All rights reserved.Cisco Public#Cisc

62、oLiveWhat is Proxy Forwarding?Why?Scaling out Endpoint LearningLeaf 1Leaf 1SpineSpineEndpoint 1Endpoint 1I am connected to Endpoint 1Send trafficdestined to EP 1 to Leaf 1Leaf 2Leaf 2Leaf 3Leaf 3Leaf 4Leaf 4Send traffic destined to an Unknown EP to any SpineOnly Leaf 1 and Spines have to program End

63、point 1 in hardwareSpines own separate anycast TEPs for mac,ipv4,and ipv6 proxy lookupsBRKDCN-390030 2023 Cisco and/or its affiliates.All rights reserved.Cisco Public#CiscoLiveHow to check the Spine-Proxy TEPleaf1#show isis dteps vrf overlay-1|grep PROXY10.0.16.65 SPINE N/A PHYSICAL,PROXY-ACAST-MAC1

64、0.0.16.64 SPINE N/A PHYSICAL,PROXY-ACAST-V410.0.16.67 SPINE N/A PHYSICAL,PROXY-ACAST-V6leaf1#show ip route vrf CL2022:vrf1192.168.0.0/24,ubest/mbest:1/0,attached,direct,pervasive*via 10.0.16.64%overlay-1,1/0,00:21:39,staticBD Subnet(Pervasive Route)next-hop should beSPINE-PROXY next-hop of Pervasive

65、 Route is IPv4 Spine Proxy TEPThree types of Spine Proxy TEP Proxy-Acast-MAC Spine-Proxy for L2 traffic(L2 Unknown Unicast mode“Hardware Proxy”)Proxy-Acast-V4 Spine-Proxy for IPv4 traffic(includes ARP Request with ARP Flooding mode“OFF”)Proxy-Acast-V6 Spine-Proxy for IPv6 trafficBRKDCN-390031 2023 C

66、isco and/or its affiliates.All rights reserved.Cisco Public#CiscoLiveWhat is COOP?COOP is the proxy-database of ACICouncil of Oracles Protocol A TCP protocol for citizens(Leafs)to publish records to oracles(Spines).Used for announcing endpoints,fabric owned IPs,multicast information,and moreSynced a

67、cross Pods/Sites with BGP EVPNEach Endpoint Record contains all information to forward(VNID,leaf TEP,mac,etc)COOP records pushed into hardware on spinesFor modular spines,scale is achieved by pushing each EP onto only two Fabric ModulesBRKDCN-390032 2023 Cisco and/or its affiliates.All rights reserv

68、ed.Cisco Public#CiscoLiveWhat if the Endpoint isnt in COOP?(ARP Glean)VRF overlay-1TEP1TEP2TEP3AnycastTEPWhat if Spines COOP DB doesnt know the destination when proxyed?X L2 Traffic:Drop L3 Traffic:ARP GleanUnicast IP1Hit Pervasive Route2Spine Proxy3No COOP entry4Encap original packet with special e

69、thertype5Flood this“Glean”to reserved multicast group6LEAFs check its BD subnets7LEAF generates ARP Request8LEAF ignores Request from Spine8If BD subnet for the unknown IP doesnt presentIf BD subnet for the unknown IP presents on LEAFBRKDCN-390033 2023 Cisco and/or its affiliates.All rights reserved

70、.Cisco Public#CiscoLiveSpine Proxy SummaryPacket coming in to LeafLEAF knowsDst MAC?Is Dst MAC onLocal Leaf?L2 or L3?LEAF has BD Subnetsfor Dst IP?LEAF knows Dst IPas EndPoint?What is BD config?FloodHardware ProxyFlood within BDSpine ProxyForward to local portForward to remote leafYesNoYesNoL2Is Dst

71、 IP onLocal Leaf?YesNoL3Forward to local portForward to remote leafYesNoSpine ProxyDst IP is L3OUT Routes?Forward to Border LeafDropYesNoIf ARP Flooding is OFF,ARPtarget-IP is used for this L3 flowBRKDCN-390034 2023 Cisco and/or its affiliates.All rights reserved.Cisco Public#CiscoLiveCapturing a Gl

72、ean with TcpdumpACI Leafs and Spines contain pseudo interfaces for traffic to and from the CPU1stGen LeafCPUkpm_inbPhysPortASICknet0knet1 EX(or Later)LeafCPUkpm_inbPhysPortASICTahoe0 For traffic going to the cpucheck knet0 and kpm_inb For traffic coming from the cpu check knet1 and kpm_inb*Note,not

73、all traffic will show up on the kpm_inb interface.However,all traffic shows on the pseudo interface*Gen1 and 2 Modular spines use psdev0,psdev1,and psdev2 interfaces.Gen 2 fixed spines use tahoe0.Gen 1 fixed spines use knet0-3 For traffic to and from the cpu check Tahoe0 and kpm_inbTraffic on the on

74、 the knet or tahoe pseudo interface will have a special ieth header.It must be decoded.Starting in 3.2 the knet_parser.py script is available on the switch cli to decodeBRKDCN-390035 2023 Cisco and/or its affiliates.All rights reserved.Cisco Public#CiscoLiveCapturing a Glean with TcpdumpGen2 or Late

75、r Leaf36BRKDCN-3900tcpdumptcpdump-xxxveixxxvei tahoe0 tahoe0-w/w/bootflashbootflash/tahoe0.pcap/tahoe0.pcapknet_parser.py knet_parser.py-file/file/bootflashbootflash/tahoe0.pcap/tahoe0.pcap-pcappcap-decoder decoder tahoetahoeFrame 111Time:2019-05-16T16:56:33.059831+00:00Header:ieth_extn CPU ReceiveC

76、PU Receivesup_qnum:0 x14,sup_code:0 x21,istack:ISTACK_SUP_CODE_SPINE_GLEAN(0 x21)Header:iethsup_tx:0,ttl_bypass:0,opcode:0 x6,bd:0 x120e,outer_bd:0 x27,dl:0,span:0,traceroute:0,tclass:0src_idx:0 x3a,src_chip:0 x0,src_port:0 x19,src_is_tunnel:1,src_is_peer:1dst_idx:0 x0,dst_chip:0 x0,dst_port:0 x0,ds

77、t_is_tunnel:0Len:148Eth:000d.0d0d.0d0d 0100.5e7f.fff1,len/ethertype:0 x8100(802.1q)802.1q:vlan:2,cos:5,len/ethertype:0 x800(ipv4)ipv4:10.0.116.64 239.255.255.241,len:130,ttl:249,id:0 x0,df:0,mf:0,offset:0 x0,dscp:32,prot:17(udp)udp:(ivxlan)0 48879,len:110ivxlan:n:1,l:1,i:1,vnid:0 x2b0000lb:0,dl:1,ex

78、ception:0,src_policy:0,dst_policy:0,src_class:0 x5c0mcast(routed:0,ingress_encap:0/802.1q),ac_bank:0,src_port:0 x0Eth:000c.0c0c.0c0c ffff.ffff.ffff,len/ethertype:0 xfff2(0 xfff2(aciaci-glean)glean)ipv4ipv4:172.16.1.1 172.16.2.2,:172.16.1.1 172.16.2.2,len:84,ttl:63,id:0 x71f9,df:1,mf:0,offset:0 x0,ds

79、cp:0,prot:1(len:84,ttl:63,id:0 x71f9,df:1,mf:0,offset:0 x0,dscp:0,prot:1(icmpicmp)icmp:echo request id:0 x9092,seq:0 x1980Traffic that Traffic that triggered Gleantriggered GleanSwitch recognizes Switch recognizes this as a Gleanthis as a GleanRX sup traffic RX sup traffic rather than TXrather than

80、TXDecode type Decode type should be should be tahoetahoe for for tahoetahoe interfaceinterfaceEgress Leaf Verification 2023 Cisco and/or its affiliates.All rights reserved.Cisco Public#CiscoLiveCapturing a Glean with TcpdumpGen1 Leaf Exampletcpdump-xxxvei knet0-w/bootflash/knet0.pcapknet_parser.py-f

81、ile/bootflash/knet0.pcap-pcap-decoder knettcpdump-xxxvei knet1-w/bootflash/knet1.pcapknet_parser.py-file/bootflash/knet1.pcap-pcap-decoder knettcpdumptcpdump-xxxveixxxvei kpm_inbkpm_inb ether proto 0 xfff2ether proto 0 xfff2a-leaf102#tcpdump-xxxvei kpm_inb ether proto 0 xfff2tcpdump:listening on kpm

82、_inb,link-type EN10MB(Ethernet),capture size 65535 bytes15:27:37.663580 00:0c:0c:0c:0c:0c(oui Unknown)Broadcast,ethertype Unknown(0 xfff2),length 94:0 x0000:ffff ffff ffff 000c 0c0c 0c0c fff2 45000 x0010:0054 aa4b 4000 3f01 825d 0404 0464 03030 x0020:0396 0800 0dc6 2384 38db 5275 dd5c 00000 x0030:00

83、00 9e35 0100 0000 0000 1011 1213 14150 x0040:1617 1819 1a1b 1c1d 1e1f 2021 2223 24250 x0050:2627 2829 2a2b 2c2d 2e2f 3031 3233knet0 would show Rx traffic(similar output as Tahoe0)knet1 would show Tx trafficNo decode necessary for kpm_inb(cpu)interfaceGleans arent easily readableEgress Leaf Verificat

84、ionBRKDCN-390037 2023 Cisco and/or its affiliates.All rights reserved.Cisco Public#CiscoLiveLayer 3 Unicast Glean ScenarioVerify ARP on Remote Leafa-leaf205#show show ipip arparp internal eventinternal event-history event|grep history event|grep-F F-B B 1 172.16.2.21 172.16.2.273)Event:E_DEBUG_DSF,l

85、ength:127,at 316928 usecs after Wed May 1 08:31:53 2019Updating Updating epmepm ifidxifidx:1a01e000 vlan:105 ip:172.16.2.2172.16.2.2,ifMode:128 mac:0000.1111.22220000.1111.222275)Event:E_DEBUG_DSF,length:152,at 316420 usecs after Wed May 1 08:31:53 2019log_collect_arp_pkt;sip=172.16.2.2172.16.2.2;di

86、p=172.16.2.254172.16.2.254;interface=Vlan104;info=Garp Check adj:(nil)77)Event:E_DEBUG_DSF,length:142,at 131918 usecs after Wed May 1 08:28:36 2019log_collect_arp_pkt;dip=172.16.2.2172.16.2.2;interface=Vlan104;iod=138;Info=Internal Request DoneInternal Request Done78)Event:E_DEBUG_DSF,length:136,at

87、131757 usecs after Wed May 1 08:28:36 2019log_collect_arp_glean;dip=172.16.2.2172.16.2.2;interface=Vlan104;info=Received pkt FabricReceived pkt Fabric-Glean:1Glean:179)Event:E_DEBUG_DSF,length:174,at 131748 usecs after Wed May 1 08:28:36 2019log_collect_arp_glean;dip=172.16.2.2172.16.2.2;interface=V

88、lan104;vrf=CiscoLive2020:vrf1;info=Address in PSVI subnet or special VIPAddress in PSVI subnet or special VIPGlean Group Range Glean Group Range included as included as BidirBidir on IPNon IPNGlean Received,Glean Received,DstDst IP IP is in BD Subnetis in BD SubnetARP Request is ARP Request is gener

89、ated by leafgenerated by leafResponse Response ReceivedReceivedEndpoint Endpoint Learn InstalledLearn InstalledEgress Leaf VerificationBRKDCN-390038How ACI Builds Forwarding Tables 2023 Cisco and/or its affiliates.All rights reserved.Cisco Public#CiscoLiveACI BehaviorLegacy BehaviorBuilding Adjacenc

90、y TablesACI combines ARP and MAC Tables into the Endpoint TableARP/ND tables map Layer 3 to Layer 2ARP/ND tables are updated by control-plane messagesMAC Address Table used for switching decisionsMac Address Table updated by dataplaneEndpoint table contains endpoints,which are Layer 2 addresses OR L

91、ayer 3 addresses OR a combination of Layer 2 and Layer 3 addressesBy default,both Layer 2 and Layer 3 information is updated by dataplaneUsed for security and forwarding policyBRKDCN-390040 2023 Cisco and/or its affiliates.All rights reserved.Cisco Public#CiscoLiveBuilding Endpoint TablesResourceEPM

92、 Endpoint ManagerSup process for managing endpoints.EPMC Endpoint Manager ClientLine card process that sits between hardware layer(HAL)and EPMHAL Hardware Abstraction LayerView of what is programmed into the ASIC.SupervisorLine CardAsicshow system internal epm endpoint mac show system internal epm e

93、ndpoint mac show system internal epm endpoint ip show system internal epm endpoint ip vsh_lc vsh_lc c“show system internal epmc endpoint mac c“show system internal epmc endpoint mac”vsh_lc vsh_lc c“show system internal epmc endpoint ip”c“show system internal epmc endpoint ip”vsh_lc vsh_lc-c show pla

94、t internal hal ep l2 mac c show plat internal hal ep l2 mac vsh_lc vsh_lc-c show plat internal hal ep l3 ip“c show plat internal hal ep l3 ip“!L3 Endpoints are put into HW Routing Tablevsh_lc vsh_lc-c show plat internal hal l3 routes|grep EPc show plat internal hal l3 routes|grep EPTable InfoCommand

95、s to VerifyEndpoints can be Endpoints can be programmed via software programmed via software process or by hardware process or by hardware dataplane learns(HAL)dataplane learns(HAL)BRKDCN-390041 2023 Cisco and/or its affiliates.All rights reserved.Cisco Public#CiscoLiveWhat about ARP?ARP Tables are

96、still used in ACI forL3outsOverlay adjacencies VXLAN Endpoints(AVE,K8s,Openstack,etc)APIC/Fabric node adjacenciesResourceAdjacency Manager.Programmed by ARP process.UFIBHAL Hardware Abstraction LayerView of what is programmed into the ASIC.SupervisorLine CardAsicshow ip arp vrf show ip arp vrf vsh_l

97、c vsh_lc c“show forwarding c“show forwarding adjacency”adjacency”vsh_lc vsh_lc-c show plat internal c show plat internal hal l3 routes”hal l3 routes”Table InfoCommands to VerifyBRKDCN-390042 2023 Cisco and/or its affiliates.All rights reserved.Cisco Public#CiscoLiveBuilding Routing TablesResourceURI

98、B/MRIB the unicast and multicast routing tables.Programmed by route protocolUFIB/MFIB the unicast and multicast forwarding tables on the Line CardHAL Hardware Abstraction LayerView of what is programmed into the ASIC.SupervisorLine CardAsicshow ip route x.x.x.x/y vrf show ip route x.x.x.x/y vrf show

99、 ip mroute x.x.x.x/y vrf show ip mroute x.x.x.x/y vrf vsh_lc vsh_lc-c show forwarding route vrf“c show forwarding route vrf“vsh_lc vsh_lc-c show forwarding multicast route vrf c show forwarding multicast route vrf vsh_lc vsh_lc-c show platform internal hal l3 routes vrf”c show platform internal hal

100、l3 routes vrf”vsh_lc vsh_lc-c show platform internal hal l3 mcast routes vrf”c show platform internal hal l3 mcast routes vrf”vsh_lc vsh_lc-c show plat internal hal l3 routes vrf|grep MCc show plat internal hal l3 routes vrf|grep MCTable InfoCommands to VerifyBRKDCN-390043 2023 Cisco and/or its affi

101、liates.All rights reserved.Cisco Public#CiscoLiveWhen Troubleshooting Layer 3 Flows AlwaysTroubleshooting TIP1)Check if there is an Endpoint LearnIf not then2)Check if there is a BD(pervasive)static routeIf not then3)Check if there is an External Routeshow endpoint ip show endpoint ip show system in

102、ternal epm endpoint ip show system internal epm endpoint ip show ip route x.x.x.x/y vrf show ip route x.x.x.x/y vrf Check Endpoint Table before Routing TableBRKDCN-390044 2023 Cisco and/or its affiliates.All rights reserved.Cisco Public#CiscoLiveProgramming ContractsResourcePolicy Manager.Programmed

103、 by leaf policy-element processACLQOSHAL Hardware Abstraction LayerView of what is programmed into the ASIC.SupervisorLine CardAsicshow zoningshow zoning-rulesrulesvsh_lc vsh_lc-c“show system internal aclqos zoningc“show system internal aclqos zoning-rules”rules”vsh_lc vsh_lc-c“show plat internal ha

104、l objects policy zoningrule”c“show plat internal hal objects policy zoningrule”Table InfoCommands to VerifyBRKDCN-390045 2023 Cisco and/or its affiliates.All rights reserved.Cisco Public#CiscoLiveHAL Hardware Abstraction LayerWouldnt it be great if there was a single point to validate forwarding and

105、 security classification?TRIEDLEFTTCAMPolicy TCAMOverflow TCAMHardware Forwarding TablesHardware Policy TablesASICvsh_lc vsh_lc-c show platform internal hal l3 routes”c show platform internal hal l3 routes”HALApplicable to EX and Later HardwareBRKDCN-390046 2023 Cisco and/or its affiliates.All right

106、s reserved.Cisco Public#CiscoLiveHAL Hardware Abstraction LayerL3 Lookup of Hardware Tablesmodule-1#show plat internal hal l3 routes vrf CL2022:vrf1-!-|LID|!|VRF|Prefix/Len|RT|Type|!|CLSS|Flags|-|-|-|-|!|-|-|-|-|-|-|!|-|-|4626|192.168.100.10/32|EP|TRIE|!|c002|le,bne,sne,dl|4626|10.99.99.0/24|UC|TCAM

107、|!|8004|sc,spi,dpi|4626|192.168.255.0/24|UC|TCAM|!|24|sc,spi,dpi,dr|4626|192.168.200.11/32|EP|TRIE|!|8003|sc,le,sne|-!-|Applicable to EX and Later HardwareConsolidated view of routes Consolidated view of routes for Endpoints,Shared for Endpoints,Shared Services,and External routesServices,and Extern

108、al routesPcTag from destination PcTag from destination EPGused for contract lookupEPGused for contract lookupMuch more info available in full output!BRKDCN-390047 2023 Cisco and/or its affiliates.All rights reserved.Cisco Public#CiscoLiveHAL Hardware Abstraction LayerL2 Lookup of Hardware Tablesmodu

109、le-1#show platform internal hal ep l2 all=BD EP L2 L2 S BdId Name T Mac IfId Ifname Class=b BD-11 Pl 00:00:11:11:22:22 1a010000 Eth1/17 c0031a BD-26 Xr 00:00:22:22:33:33 18010004 Tunnel4 400f21 BD-33 Pl 00:00:22:22:33:33 16000002 Po3 4002Applicable to EX and Later HardwareConsolidated view of all Co

110、nsolidated view of all learned Mac Addresseslearned Mac AddressesPcTag from destination PcTag from destination EPGused for contract lookupEPGused for contract lookupMuch more info available in full output!BRKDCN-390048Understanding the Configuration Options 2023 Cisco and/or its affiliates.All right

111、s reserved.Cisco Public#CiscoLiveVRF Level Forwarding OptionsFeatureWhat Does it Do?Policy Control Enforcement PreferenceIf disabled,policy is never applied between EPGs.If enabled,contracts are enforced.IP Dataplane LearningIf Disabled,ACI uses legacy behavior for learning endpoints.Layer 3 endpoin

112、ts are learned by ARP/GARP/ND and Layer 2 endpoints are learned by dataplane.Policy Control Enforcement DirectionIf set to Ingress,contract enforcement for l3out flows is done on service leaf.Egress enables enforcement on Border Leaf(requires remote learning to be enabled)SLeafSLeafBLeafBLeafL3outIn

113、gress leaf sets policy applied bitsEgress leaf does not set policy applied bitsIngress EnforcementSLeafSLeafBLeafBLeafL3outIngress leaf does not set policy applied bitsEgress leaf sets policy applied bitsEgress EnforcementBRKDCN-390050 2023 Cisco and/or its affiliates.All rights reserved.Cisco Publi

114、c#CiscoLiveBridge-Domain Level Forwarding OptionsFeatureWhat Does it Do?L3 Unknown Multicast FloodingFor non-link-local L3 multicast traffic in a PIM-disabled BD,should a leaf with no snooping entries flood in BD(flood)or wait for joins(OMF)?Multidestination FloodingFor L2 mcast and broadcast,flood,

115、drop,or flood within epg encap?If flooding with EPG encap,proxy-arp is required for cross-epg L2 communicationL2 Unknown UnicastIf destination mac is unicast and unknown,flood or proxy to spines?Proxied,L2 Unknown Unicast is dropped if the Destination MAC isnt known in COOPBRKDCN-390051 2023 Cisco a

116、nd/or its affiliates.All rights reserved.Cisco Public#CiscoLiveBridge-Domain Level Forwarding OptionsFeatureWhat Does it Do?Limit IP Learning to SubnetOnly learn IPs if they are within the configured BD subnet for local learns.Unicast RoutingEnable IP learning as well as unicast routing(if a BD subn

117、et is configured)IP Data-plane LearningConfigured underneath the BD subnet.When disabled,IP/IPv6 learning is done via ARP/NDARP FloodingWhen disabled,ARP is unicast routed based on the Target IP(if known)Who has Who has 192.168.100.11?192.168.100.11?leaf#show endpoint ip 192.168.100.11show endpoint

118、ip 192.168.100.11leaf#show ip route 192.168.100.11 vrf CL2022:vrf1show ip route 192.168.100.11 vrf CL2022:vrf1192.168.100.0/24,ubest/mbest:1/0,direct,pervasivepervasive*via 10.0.176.66%overlay-1,1/0,01w00d,staticrecursive next hop:10.0.176.66/32%overlay-1Proxy!Proxy!BRKDCN-390052 2023 Cisco and/or i

119、ts affiliates.All rights reserved.Cisco Public#CiscoLiveEPG Level Forwarding OptionsFeatureWhat Does it Do?Flood in EncapsulationFeature is enabled for just the EPG(rather than all epgs in the BD).Requires proxy arp for L2 traffic between encaps.L4-L7 Virtual IPsDesigned for Direct Server Return flo

120、ws.This disables dataplane learning per IP.IP is learned by ARP/ND.Disable DP Learning Per-IP/PrefixDisables dataplane learning.More specific than VRF-level option.In most cases should be used for DSR too.New in 5.2,can also be configured on BDBRKDCN-390053 2023 Cisco and/or its affiliates.All right

121、s reserved.Cisco Public#CiscoLiveGlobal Forwarding OptionsFeatureWhat Does it Do?Enforce Subnet CheckDont learn an IP(both local and remote)if it is not within a configured BD subnet in the VRF.Disable Remote EP Learning on BLsRemote IP learning is disabled for Unicast flows on a leaf in a specific

122、VRF if an l3out exists in the same VRF Multicast sources are still learnedAlso implicitly disabled when intersite l3out is configuredBRKDCN-390054The Anatomy of an ACI Switch 2023 Cisco and/or its affiliates.All rights reserved.Cisco Public#CiscoLiveLEAF ASIC GenerationsCisco ASIC1stgenerationBroadc

123、omCPU2ndgeneration(or later)CPUGSTingressLSTingressGSTegressLSTegressFPTilesN9K-C9332PQN9K-C9372PXN9K-C9372PX-EN9K-C9372TXN9K-C9372TX-EN9K-C9396PXN9K-C9396TXN9K-C93120TXN9K-C93128TXN9K-C*-EXN9K-C*-FXN9K-C*-FX2N9K-C*-FX3 LST:Local Station Table,GST:Global Station Table FP Tile:Forwarding and Policy T

124、ileTo SPINETo SPINECloud Scale ASICLocal EP LearnDest EP LookupRemote EP LearnDest EP LookupComplete separation of+Ingress and Egress+Source Learn and Destination LookupSeparate GST/LST for IP and MACMore flexible/scalable with configurable tilesAbstracted with HAL Tile X for both source learn and d

125、estination lookupTile X:IPTile Y:MACetc.N9K-C*-FXPN9K-C*-GXN9K-C*-GX2BRKDCN-390056 2023 Cisco and/or its affiliates.All rights reserved.Cisco Public#CiscoLiveCisco ASICCloud Scale ASICLine cardFabric cardSUPCPUCloud Scale ASICSPINE ASIC GenerationsBroadcom1stgenerationCisco ASIC2ndgeneration(or late

126、r)COOP DatabaseN9K-X9736PQN9K-C9504-FMN9K-C9508-FMN9K-C9516-FMN9K-*XN9K-C*FM-EN9K-C*FM-E2N9K-C*FM-G number of ASIC per card depends on modelLine cardFabric cardSUPCPUCisco ASICN9K-*CN9K-*XN9K-C9336PQCOOP DatabaseTEP InformationTEP InformationLine cardBox spineFabric cardLine cardBox spineFabric card

127、BRKDCN-390057 2023 Cisco and/or its affiliates.All rights reserved.Cisco Public#CiscoLiveInside an ACI Switch ASIC(Gen 2 and Later)PhysPortParser Block(PRX)Lookup Block(LUA LUB)Forwarding Block(FPA FPC)Lookup Block(LUC)ACL Engine(ACA ACC)Lookup Block(LUD)Load Balancing(LBX)Buffering and Queueing Eng

128、ine(BMX/QSX/BAX)Rewrite Block(RWX)PhysPortEvaluate frame formatBased on FP result,re-evaluate LU ResultLookup Destination IP/MAC;FPC determines contractDetermine VRF,VLAN,EPG,etcEvaluate any matching ACL entriesRevaluate LU Result based on ACL hitsCalaculate Load-Balance HashingBuild the final frame

129、(egress qtag,etc)Apply QoS and Buffering PoliciesBRKDCN-390058 2023 Cisco and/or its affiliates.All rights reserved.Cisco Public#CiscoLiveInside an ACI Modular SpineASIC 0ASIC 1ASIC 2ASIC 3Ports A-BPorts G-HPorts E-FPorts C-D number of ASIC per card depends on modelACI Line CardFabric ModuleFabric M

130、oduleFabric ModuleFabric ModuleASIC 0ASIC 0ASIC 0ASIC 0Internal PCs(2 ports per)sp#vsh vsh-c slot 2 show plat internal hal l3 routes”c slot 2 show plat internal hal l3 routes”sp#vsh vsh-c slot 26 show plat internal hal l3 routes”c slot 26 show plat internal hal l3 routes”40.0.99.139/323.124.199.13/3

131、20.156.151.177/32Where are the linecard Where are the linecard forwarding tables?forwarding tables?What are the strange IPs on What are the strange IPs on the Fabric Modules?the Fabric Modules?BRKDCN-390059 2023 Cisco and/or its affiliates.All rights reserved.Cisco Public#CiscoLiveInside an ACI Modu

132、lar SpineHow is traffic forwarded?For Proxied TrafficFor Proxied TrafficDepending on if the dest IP is the L2 or L3 Proxy TEP the VRF VNID+Dest IP OR BD VNID+Dest MAC is used to hash a synthetic Dest IP and VRF IDSynthetic information is used on LC to hash the uplink port to FMFM routing lookup is b

133、ased on Synthetic IPEach Synthetic IP is owned by two FMsFM uses vnTag to tell egress LC which front panel port to useBRKDCN-390060 2023 Cisco and/or its affiliates.All rights reserved.Cisco Public#CiscoLiveInside an ACI Modular SpineHow is traffic forwarded?For Transit TrafficFor Transit TrafficLin

134、e card hashes across ALL FM uplinksALL FMs have overlay TEP routesFM uses vnTag to tell egress LC which front panel port to useBRKDCN-390061 2023 Cisco and/or its affiliates.All rights reserved.Cisco Public#CiscoLiveInside an ACI Modular SpineASIC 0ASIC 1ASIC 2ASIC 3Ports A-BPorts G-HPorts E-FPorts

135、C-DACI Line CardFabric ModuleFabric ModuleFabric ModuleFabric ModuleASIC 0ASIC 0ASIC 0ASIC 0Internal PCs(2 ports per)sp#moquery-c coopIpv4Rec-f coop.Ipv4Rec.addr=10.0.0.10#coop.Ipv4Recaddr :10.0.0.10synthIp :15.180.164.253synthVrf :250Ingress LC hashes Synth IP and Ingress LC hashes Synth IP and VRF

136、 based on Real Tenant IP+VRFVRF based on Real Tenant IP+VRFsp#vsh-c slot 26 show forwarding route platform|grep-A 10 15.180.164.253!Table:(IN-HW)Type=100 Vrf=750 Synth=0!FC Cards/ASICs:FC22/ASIC-0 FC26/ASIC-0Synthetic info programmed Synthetic info programmed on FMs 22 and 26on FMs 22 and 261DIPO is

137、 IPv4 Proxy TEPDIPI is 10.0.0.10VRF Vnid is 111111 Ingress Traffic:BRKDCN-390062 2023 Cisco and/or its affiliates.All rights reserved.Cisco Public#CiscoLiveInside an ACI Modular SpineASIC 0ASIC 1ASIC 2ASIC 3Ports A-BPorts G-HPorts E-FPorts C-DACI Line CardFabric ModuleFabric ModuleFabric ModuleFabri

138、c ModuleASIC 0ASIC 0ASIC 0ASIC 0Traffic Hashes across either FM 22 Traffic Hashes across either FM 22 or 26 uplinksor 26 uplinksDIPO is IPv4 Proxy TEPDIPI is 10.0.0.10VRF Vnid is 111111 Fabric Module does route lookup Fabric Module does route lookup based on Synthetic IP and VRFbased on Synthetic IP

139、 and VRFvnTag is derived from route lookup,vnTag is derived from route lookup,FM forwards to egress LC which FM forwards to egress LC which forward only based on vnTagforward only based on vnTagEgress LC knows exactly which frontEgress LC knows exactly which front-panel port to panel port to forward

140、 out of based on the received vntagforward out of based on the received vntag2345Ingress Traffic:Internal PCs(2 ports per)BRKDCN-390063Understanding the Tools 2023 Cisco and/or its affiliates.All rights reserved.Cisco Public#CiscoLiveStart with High-level ToolsUse Endpoint Tracker for Building a Top

141、ologyEP Locally Learned on EP Locally Learned on pod 2,nodes 401pod 2,nodes 401-402402No EP Learn,is this an No EP Learn,is this an L3out?L3out?BRKDCN-390065 2023 Cisco and/or its affiliates.All rights reserved.Cisco Public#CiscoLiveStart with High-level ToolsUse Atomic Counters to Check for Overlay

142、 Drops and Latency(PTP)BRKDCN-390066 2023 Cisco and/or its affiliates.All rights reserved.Cisco Public#CiscoLiveStart with High-level ToolsUse Atomic Counters to Check for Overlay Drops and Latency(PTP)104 Microseconds of 104 Microseconds of delay in overlaydelay in overlayNo overlay drops!No overla

143、y drops!BRKDCN-390067 2023 Cisco and/or its affiliates.All rights reserved.Cisco Public#CiscoLiveStart with High-level ToolsUse Tenant Visibility tools to check for Contract DropsThis flow is being This flow is being contract droppedcontract droppedapic4#show acllog deny l3 pkt tenant common vrfCORE

144、srcIp dstIp protocol srcPort dstPort node srcIntf vrfEncap-BRKDCN-390068 2023 Cisco and/or its affiliates.All rights reserved.Cisco Public#CiscoLiveStart with High-level ToolsPort Counters are as Useful as Everleaf1#show interface eth1/8Ethernet1/8 is upadmin state is up,Dedicated InterfaceLast link

145、 flapped 03:07:41RX3527922 unicast packets !ommitted4041582 input packets 609518993 bytes12 jumbo packets 0 storm suppression bytes0 runts 0 giants 0 CRC 0 Stomped CRC0 no buffer0 input error 0 short frame 0 overrun!ommitted0 watchdog 0 bad etype drop 0 bad proto drop!ommitted0 input with dribble 0

146、input discard0 input buffer drop 0 input total dropTX32262479565 unicast packets !ommitted32395063346 output packets 49034781261665 bytes32249687943 jumbo packets0 output error 0 collision 0 deferred 0 late collision0 lost carrier 0 no carrier 0 babble 0 output discard0 output buffer drops 0 output

147、total dropsFrames received with bad FCSIndicates a previously stomped frame was receivedWhat is a Stomp?When a frame is received with a bad FCS and/or is malformedANDThe frame is cut-through switchedThe switch will invert the new CRC to tell the first store-and-forward device to drop itFrame transmi

148、tted with stomped CRCBuffer drops,sign of congestionBRKDCN-390069 2023 Cisco and/or its affiliates.All rights reserved.Cisco Public#CiscoLiveStart with High-level ToolsUsing moquery to check port counters fabric-wide#Check Fabric#Check Fabric-wide for FCS Errorswide for FCS Errorsmoquery-c rmonDot3S

149、tats-f rmon.Dot3Stats.fCSErrors=1|egrep dn|fCSErrors”#Check Fabric#Check Fabric-wide for total CRC Stomp+FCS Errorswide for total CRC Stomp+FCS Errorsmoquery-c rmonEtherStats-f rmon.EtherStats.cRCAlignErrors=1|egrep dn|cRCAlignErrors”#Check Fabric#Check Fabric-wide for Output Buffer Dropswide for Ou

150、tput Buffer Dropsmoquery-c rmonEgrCounters-f rmon.EgrCounters.bufferdroppkts=1|egrep dn|bufferdroppkts”#Check Fabric#Check Fabric-wide Output Errorswide Output Errorsmoquery-c rmonIfOut-f rmon.IfOut.errors=1|egrep dn|errors”BRKDCN-390070 2023 Cisco and/or its affiliates.All rights reserved.Cisco Pub

151、lic#CiscoLiveELAM Embedded Logic Analyzer ModuleIt is a tripwire in hardwareThe first frame to match a specified condition trips itReport is created with vast amount of data regarding asic decisionsDst TCP 10.0.0.1:3000Dst Dst TCP 10.0.0.1:3001TCP 10.0.0.1:3001Dst TCP 10.0.0.1:3002vsh_lcdebug platfo

152、rm internal tah elam asic 0trigger resettrigger init in-select 6 out-select 1set outer ipv4 dst_ip 10.0.0.1set outer l4 dst-port 3001startmodule-1(DBG-elam-insel6)#statELAM STATUS=Asic 0 Slice 0 Status ArmedAsic 0 Slice 1 Status Triggeredmodule-1(DBG-elam-insel6)#ereport|grep drop reasonRW drop reas

153、on :no dropLU drop reason :no dropMatching frame was Matching frame was caught!caught!Frame was not dropped in lookups!BRKDCN-390071 2023 Cisco and/or its affiliates.All rights reserved.Cisco Public#CiscoLiveWhat ASIC should be set in the ELAM?vsh_lcdebug platform internal elam asic 0ModelRoleAsic f

154、or ElamN9K-C*CFixed SpinerocN9K-C*GXFixed SpineappN9K-C*-EXLeaftahN9K-C*-FX/FXP/FX2LeafrocN9K-C*-GXLeafappN9K-C*-GX2LeafchoN9K-X97*-EXSpine LCtahN9K-X97*-FXSpine LCrocN9K-X97*-GXSpine LCappN9K-C95*-FM-ESpine FMtahN9K-C950*-FM-E2Spine FMrocN9K-C95*-FM-GSpine FMappBRKDCN-390072 2023 Cisco and/or its a

155、ffiliates.All rights reserved.Cisco Public#CiscoLiveSteps to Using Elam on Gen2+Leaf or Fixed Spinevsh_lcdebug platform internal tah elam asic 0trigger resettrigger init in-select 6 out-select 0set outer ipv4 dst_ip 10.0.0.1set outer l4 dst-port 3001startElams are run from Elams are run from the lin

156、e card shellthe line card shellRefer to“What ASIC should Refer to“What ASIC should be set in the ELAM”slide be set in the ELAM”slide Leafs and fixed spines are single Leafs and fixed spines are single asic switches.Always use asic 0asic switches.Always use asic 0Failing to reset the trigger Failing

157、to reset the trigger can cause past elam can cause past elam configurations to take effect.configurations to take effect.Always reset the trigger!Always reset the trigger!module-1(DBG-elam)#trigger init in-select?!ommitted14 Outer(l2(vntag)|l3|l4)-inner(l2|l3|l4)-ieth6 Outerl2-outerl3-outerl47 Inner

158、l2-innerl3-innerl4!ommittedDetermines which headers conditions can be matched in.Use 14 or 7 when matching vxlan encapsulated headers.Use 0 or 1Use 0 or 1BRKDCN-390073 2023 Cisco and/or its affiliates.All rights reserved.Cisco Public#CiscoLiveSteps to Using Elam on Gen2+Leaf or Fixed Spinevsh_lcdebu

159、g platform internal tah elam asic 0trigger resettrigger init in-select 6 out-select 0set outer ipv4 dst_ip 10.0.0.1set outer l4 dst-port 3001startUse“Use“set outer”or”or“set inner”depending”depending on inon in-select and if select and if matching outer or inner matching outer or inner headers in vx

160、lan packetheaders in vxlan packetWhich headers to match Which headers to match conditions for?conditions for?What to match in the What to match in the header?header?Finally enable the elam!Finally enable the elam!When running When running stat if if Triggered is seen,this is seen,this means a matchi

161、ng packet was receivedmeans a matching packet was receivedBRKDCN-390074 2023 Cisco and/or its affiliates.All rights reserved.Cisco Public#CiscoLiveReading an ElamAt a high-levelmodule-1(DBG-elam-insel6)#ereport!ommitted-Outer L3 Header-L3 Type :IPv4IP Version :4DSCP :0IP Packet Length :84(=IP header

162、(28 bytes)+IP payload)Dont Fragment Bit :setTTL :64IP Protocol Number :ICMPDestination IP :192.168.200.11Source IP :192.168.100.10!omittedContract ResultContract Drop :noContract Logging :noContract Applied :yesContract Hit :yesereportereport available since 4.2ereport provides a simple,human-readab

163、le report outputereport requires=5.2 code for modular spinesGroups data into outer/inner,headers,and lookup resultsBRKDCN-390075 2023 Cisco and/or its affiliates.All rights reserved.Cisco Public#CiscoLiveAt a low-levelreport detail|grep F-|grep-v VECTOR|grep-v endLU BEGIN-LUA-LUB-LUC-LUD-LU END-*FP

164、latch results-*LBX latch results -*ACX latch results -RW BEGIN-RW END-ereportereport available since 4.2An elam report provides a walkthrough of each ASIC blockEach decision in each block is recordedRefer to“Inside an ACI Switch ASIC”from part 1 for more detailsAll output is in HEXReading an ElamBRK

165、DCN-390076 2023 Cisco and/or its affiliates.All rights reserved.Cisco Public#CiscoLiveWhat if Elam Shows a Drop?ereportereport available since 4.2ereportLookup Drop-LU drop reason :SECURITY_GROUP_DENYDrop CodeWhat Does it Mean?What to Do?ACL_DROPFor traffic destined to the CPU on an FX switch it is

166、expected and cosmetic.Also seen when traffic was received from a fabric port and the leaf has a remote EP learn with no bounce flag.Ignore if its an FX switch and destined to local switch IP/process.Otherwise,check for incorrect EP learn.DCI_*_XLATE_MISSFor multisite/remote-leaf,there was no matchin

167、g vnid or pctag translation found.Check contracts between local and remote resources.INFRA_ENCAP_SRC_TEP_MISSNo route and/or tunnel found back to the outer source IPCheck for a tunnel pointing back to the outer source IP.Also,check for a route in overlay.SECURITY_GROUP_DENYSECURITY_GROUP_DENYFrame w

168、as contract droppedFrame was contract droppedMake sure a contract is configured to allow Make sure a contract is configured to allow the flow.the flow.SRC_VLAN_MBRReceived vlan not programmed on ingress port.Check if the frame was correct tagged/untagged.Make sure no invalid-path faults exist for th

169、e epg.UC_PC_CFG_TABLE_DROPNo route was found for the destination.Check the routing table for the destination.VLAN_XLATE_MISSReceived vlan doesnt exist on the switch.Check if the frame is tagged with correct vlan.Check for invalid-path faults on the epg.Common Drop ReasonsCommon Drop ReasonsBRKDCN-39

170、0077 2023 Cisco and/or its affiliates.All rights reserved.Cisco Public#CiscoLiveSteps to Using Elam on Gen2+Modular SpineChallenges of Modular SpinesLine cards(and potentially FMs)have multiple asicsElam must specify asic numberIngress/Egress ports may be internal LC FM connectionsereport only avail

171、able in 5.2 and laterFortunately,spine Fortunately,spine elamselams arent needed as commonly as leaf arent needed as commonly as leaf elamselams!BRKDCN-390078 2023 Cisco and/or its affiliates.All rights reserved.Cisco Public#CiscoLiveAutomating Modular Spine ELAMsCLI-based Modular Spine Elam tool av

172、ailable at EasySpineElamspine1#./easy-spine-elam.sh-m all-d ingressFinal module list is:2 23 26 32022-06-08T14:55:57 In-select-14 and out-select-0 are being used.!ommitted70.inner ipv4 destination ip Format:d.d.d.d71.inner ipv4 protocol Format:0-25573.inner ipv4 source ip Format:d.d.d.d91.inner l4 d

173、est port Format:0-65535Select corresponding numbers of conditions to set.Separate numbers with commas.Ex:1,2,3,4,5Enter selections:70,73,71,91Enter inner ipv4 destination ip Format:d.d.d.d:80.0.0.1Enter inner ipv4 source ip Format:d.d.d.d:150.0.0.100Enter inner ipv4 protocol Format:0-255:6Enter inne

174、r l4 dest port Format:0-65535:8989Easily Set Conditions on All or Some ModulesWhich conditions to match?Which conditions to match?Set conditionsSet conditionsBRKDCN-390079 2023 Cisco and/or its affiliates.All rights reserved.Cisco Public#CiscoLiveAutomating Modular Spine ELAMsCLI-based Modular Spine

175、 Elam tool available at EasySpineElam2022-06-08T14:56:28 Checking elam status for module 22022-06-08T14:56:28 Checking elam status for module 232022-06-08T14:56:28 Checking elam status for module 262022-06-08T14:56:28 Checking elam status for module 3ELAM TRIGGERED on module 26:ASIC:0 SLICE:1ELAM TR

176、IGGERED on module 2:ASIC:3 SLICE:1Type status to check elam status again.Type ereport,report or report detail to collect all reports:ereport2022-06-08T14:57:36 Collecting report for module 26 asic 0.2022-06-08T14:57:36 Collecting report for module 2 asic 3.2022-06-08T14:57:46 Converting reports to e

177、report format!The following decoded elams are available-/data/techsupport/mod26-asic0-elamreport-2022-06-08T14-57-36-EREPORT/data/techsupport/mod2-asic3-elamreport-2022-06-08T14-57-36-EREPORT2022-06-08T14:57:49 FINISHED!Generate and view ereport from all Triggered Modules!ELAM triggered on ELAM trig

178、gered on LC and FM!LC and FM!Locally view or copy Locally view or copy off the final ereportsoff the final ereportsBRKDCN-390080 2023 Cisco and/or its affiliates.All rights reserved.Cisco Public#CiscoLiveShouldnt ELAM be More Simple?https:/ELAM(Embedded Logic Analyzer Module)Perform an ASIC level pa

179、cket captureELAM Assistant You can perform ELAM like a TAC engineer!With a nicely formatted result reportDetail Explanations:https:/ to use video,pictures A download link for ELAM AssistantElam Assistant in DCAppCenterBRKDCN-390081 2023 Cisco and/or its affiliates.All rights reserved.Cisco Public#Ci

180、scoLiveELAM Assistant in ACI AppCenter(example)1.Perform an ElamTriggered!andReport is ReadySet ParametersBRKDCN-390082 2023 Cisco and/or its affiliates.All rights reserved.Cisco Public#CiscoLiveELAM Assistant in ACI AppCenter(example)2.Read a ReportScroll DownClick to see reportReport shows up here

181、BRKDCN-390083 2023 Cisco and/or its affiliates.All rights reserved.Cisco Public#CiscoLiveFTRIAGE Automating Elamsapic1#ftriage route-ii LEAF:101,102-dip 10.99.99.100-sip 192.168.100.1020:19:54 INFO main:1295 L3 packet Seen on leaf102 Ingress:Eth1/34(Po5)Egress:Eth1/54 Vnid:252313620:19:55 INFO main:

182、1364 leaf102:Packets egress outer SIP:10.0.176.67,DIP:10.0.64.7020:19:55 INFO main:1371 leaf102:Outgoing packets Vnid:252313620:19:56 INFO main:353 Computed ingress encap string vlan-350120:20:03 INFO main:464 Ingress BD(s)CL2022:bd120:20:03 INFO main:476 Ingress Ctx:CL2022:vrf1 Vnid:2523136!20:21:4

183、6 INFO main:1295 L3 packet Seen on spine1005 Ingress:Eth1/1 Egress:Eth1/3 Vnid:252313620:22:38 INFO fib:737 spine1005:Transit in spine20:23:32 INFO main:1295 L3 packet Seen on leaf103 Ingress:Eth1/29 Egress:Eth1/27/4 Vnid:NULL!20:24:02 INFO fib:219 leaf103:L3 out interface Ethernet1/27/420:24:10 INF

184、O main:781 Computed egress encap string vlan-105520:24:17 INFO main:1796 Packet is Exiting fabric with peer-device:N3K-1 and peer-port:Ethernet1/31Orchestrate EndOrchestrate End-toto-End End ELAMs from the APIC!ELAMs from the APIC!BRKDCN-390084 2023 Cisco and/or its affiliates.All rights reserved.Ci

185、sco Public#CiscoLiveSPAN/ERSPANDont neglect old friends!Both local span and erspan supportedERSPAN requires an l3 endpoint learned anywhere in the fabricStill the best tool for checking Packet contentsFrame formatRetransmissionsand anything else that can be seen in a pcapBRKDCN-390085 2023 Cisco and

186、/or its affiliates.All rights reserved.Cisco Public#CiscoLiveOther Tools Requiring External ResourcesNetflowCaptures flow information based on specified criteriaUseful for troubleshooting packet loss and latencyFlow TelemetryHardware directly streams flow data to Nexus Dashboard InsightsUseful for t

187、roubleshooting packet loss and latencyLatency measurements leverage PTP for additional accuracyNDI can perform additional flow analyticsBRKDCN-390086Debugging ACI BUM Flows 2023 Cisco and/or its affiliates.All rights reserved.Cisco Public#CiscoLiveARP Ingress LeafLeafLeafLeafLeafSpineSpineSpineSpine

188、EP2EP2192.168.100.11/24192.168.100.11/240000.aaaa.bbbb0000.aaaa.bbbbLeafLeafLeaf Floods ARP in BD GIPO Leaf Floods ARP in BD GIPO on selected FTAG tree with BD on selected FTAG tree with BD vnid setvnid setBridge Domain Settings:Unicast Routing DisableARP Flooding EnabledEP1EP1192.168.100.10/24192.1

189、68.100.10/240000.cccc.dddd0000.cccc.ddddARP:Who has 192.168.100.10?show ip mroute 225.0.2.128 vrf overlay-1IP Multicast Routing Table for VRF overlay-1(*,225.0.2.128/32),uptime:22w2d,isisIncoming interface:Null,RPF nbr:0.0.0.0Outgoing interface list:(count:2)Ethernet1/29.9,uptime:8w2dEthernet1/30.10

190、,uptime:22w2dCheck GIPO Route1BRKDCN-390088 2023 Cisco and/or its affiliates.All rights reserved.Cisco Public#CiscoLiveARP How to Find the GiPoFrom the GUImoquery-c fvBD-f fv.BD.dn*tn-CL2022/BD-bd1#fv.BDarpFlood :yesbcastP :225.0.2.128dn :uni/tn-CL2022/BD-bd1moquery-c l2BD-f l2.BD.name=CL2022:bd1-x

191、rsp-subtree=full rsp-subtree-class=fmcastGrp#fmcast.Grpaddr :225.0.2.128dn :sys/ctx-vxlan-2523136/bd-vxlan-14811121/fmgrp-225.0.2.128rn :fmgrp-225.0.2.128From the APIC CLIFrom the Switch CLIBRKDCN-390089 2023 Cisco and/or its affiliates.All rights reserved.Cisco Public#CiscoLiveARP Ingress LeafLeafL

192、eafLeafLeafSpineSpineSpineSpineEP2EP2192.168.100.11/24192.168.100.11/240000.aaaa.bbbb0000.aaaa.bbbbLeafLeafLeaf Floods ARP in BD GIPO Leaf Floods ARP in BD GIPO on selected FTAG tree with BD on selected FTAG tree with BD vnid setvnid setBridge Domain Settings:Unicast Routing DisableARP Flooding Enab

193、ledEP1EP1192.168.100.10/24192.168.100.10/240000.cccc.dddd0000.cccc.ddddARP:Who has 192.168.100.10?vsh_lcdebug plat internal app elam asic 0trigger resettrigger init in-select 6 out-select 0set outer arp source-ip 192.168.100.11set outer arp target-ip 192.168.100.10start!statELAM STATUS=Asic 0 Slice

194、0 Status ArmedAsic 0 Slice 1 Status ArmedAsic 0 Slice 2 Status TriggeredAsic 0 Slice 3 Status ArmedELAM the ARP request!1BRKDCN-390090 2023 Cisco and/or its affiliates.All rights reserved.Cisco Public#CiscoLiveOuter L2 Header -Access Encap VLAN :3502(0 xDAE)Outer L3 Header -ARP Opcode :Request(0 x1)

195、ARP Sender IP :192.168.100.11ARP Target IP :192.168.100.10Contract Result -Contract Drop :noContract Applied :noFINAL FORWARDING LOOKUP-Bits set in Final Forwarding Block:IFABRIC_IG MC TENANT MYTEP BRIDGE MISS FLOODLookup Drop-LU drop reason :no dropARP Ingress Leaf Elam Results(ereport)Bridge Domai

196、n Settings:Unicast Routing DisableARP Flooding EnabledNot Dropped in lookups!Not Dropped in lookups!Make sure this matches Make sure this matches what is expectedwhat is expectedFrame is flooded in the Bridge Domain!BRKDCN-390091 2023 Cisco and/or its affiliates.All rights reserved.Cisco Public#Cisc

197、oLiveARP How to Find the FTAGNo other way than Elammodule-1(DBG-elam-insel6)#ereport|grep nopad.ftagwol_lu2ba_sb_info.mc_info.mc_info_nopad.ftag:0 x8Selected ftag is 0 x8Selected ftag is 0 x8Leaf forwards to root port and OIFs for ftag 8Since GIPO is 225.0.2.128,Dest multicast address is 225.0.2.136

198、(gipo+ftag)Check ftag topology with show isis internal mcast routes ftagleaf103#show isis internal mcast routes ftagIS-IS process:isis_infraVRF:defaultFTAG Routes=FTAG ID:8 Enabled Cost:(1/6/0)-Root port:Ethernet1/29.9OIF List:Leaf appends ftag to gipo and Leaf appends ftag to gipo and forwards out

199、Eth1/29 to spineforwards out Eth1/29 to spineBRKDCN-390092 2023 Cisco and/or its affiliates.All rights reserved.Cisco Public#CiscoLiveARP-SpineLeafLeafLeafLeafSpineSpineSpineSpineEP2EP2192.168.100.11/24192.168.100.11/240000.aaaa.bbbb0000.aaaa.bbbbLeafLeafRoot spine for ftag 8 Root spine for ftag 8 f

200、orwards out OIFsforwards out OIFsBridge Domain Settings:Unicast Routing DisableARP Flooding EnabledEP1EP1192.168.100.10/24192.168.100.10/240000.cccc.dddd0000.cccc.ddddARP:Who has 192.168.100.10?spine1005#show isis internal mcast routes ftagIS-IS process:isis_infraVRF:defaultFTAG Routes=FTAG ID:8 Roo

201、t Enabled Cost:(0/0/0)-Root port:-OIF List:Ethernet1/1.20Ethernet1/2.21Ethernet1/3.19This spine is the root!Root for Root for Ftag 8Ftag 8Spine forwards out OIFs2BRKDCN-390093 2023 Cisco and/or its affiliates.All rights reserved.Cisco Public#CiscoLiveARP Egress LeafLeafLeafLeafLeafSpineSpineSpineSpi

202、neEP2EP2192.168.100.11/24192.168.100.11/240000.aaaa.bbbb0000.aaaa.bbbbLeafLeafBridge Domain Settings:Unicast Routing DisableARP Flooding EnabledEP1EP1192.168.100.10/24192.168.100.10/240000.cccc.dddd0000.cccc.ddddARP:Who has 192.168.100.10?leaf102#show vlan|grep CL2022:bd170CL2022:bd1 activeleaf102#s

203、how vlan id 70 extendedVLAN Name Encap Ports-70 CL2022:bd1 vxlan-14811121 Eth1/33,Eth1/34,Eth1/43,Eth1/48,Po5,Po6,Po8,Po9Flood out these ports and encaps in this BDEgress leaf(s)floods out Egress leaf(s)floods out all ports in the BD(except all ports in the BD(except with flood in encap)with flood i

204、n encap)3BRKDCN-390094 2023 Cisco and/or its affiliates.All rights reserved.Cisco Public#CiscoLiveARP Egress LeafLeafLeafLeafLeafSpineSpineSpineSpineEP2EP2192.168.100.11/24192.168.100.11/240000.aaaa.bbbb0000.aaaa.bbbbLeafLeafBridge Domain Settings:Unicast Routing DisableARP Flooding EnabledEP1EP1192

205、.168.100.10/24192.168.100.10/240000.cccc.dddd0000.cccc.ddddARP:Who has 192.168.100.10?Egress leaf(s)floods out Egress leaf(s)floods out all ports in the BD(except all ports in the BD(except with flood in encap)with flood in encap)vsh_lcdebug plat internal tah elam asic 0trigger resettrigger init in-

206、select 14 out-select 1set inner arp source-ip 192.168.100.11 set inner arp target-ip 192.168.100.10set inner l2 dst_mac ffff.ffff.ffffstartstatELAM STATUS=Asic 0 Slice 0 Status TriggeredAsic 0 Slice 1 Status ArmedELAM the ARP request!3BRKDCN-390095 2023 Cisco and/or its affiliates.All rights reserve

207、d.Cisco Public#CiscoLiveOuter L3 Header-Destination IP :225.0.2.136Inner L3 Header-ARP Sender IP :192.168.100.11ARP Target IP :192.168.100.10Outer L4 Header-VRF or BD VNID :14811121(0 xE1FFF1)Contract Result-Contract Drop :noFINAL FORWARDING LOOKUP-Bits set in Final Forwarding Block:IFABRIC_EG MC IN

208、FRA ENCAP MYTEP BRIDGE MISS FLOODLookup Drop-LU drop reason :no dropARP Egress Leaf Elam Results(ereport)Bridge Domain Settings:Unicast Routing DisableARP Flooding EnabledNot Dropped in lookups!Not Dropped in lookups!Destination is GIPO Destination is GIPO(225.0.2.128)+FTAG(0 x8)(225.0.2.128)+FTAG(0

209、 x8)Frame is flooded in the Bridge Domain!2023 Cisco and/or its affiliates.All rights reserved.Cisco Public#CiscoLiveARP Egress Leaf Port is VPCBridge Domain Settings:Unicast Routing DisableARP Flooding EnabledBoth VPC members receive a flooded copyOne VPC member is the Designated Forwarder(DF)for t

210、he flowDF is hashed per flowOnly DF floods out VPC interfacesmodule-1(DBG-elam-insel14)#ereport|grep df|grep vpcsug_lub_latch_results_vec.lub4_1.vpc_df:0 x0sug_fpx_lookup_vec.lkup.dciptvec.pt.vpc_df:0 x0sug_fpc_lookup_vec.fplu_vec.lkup.dciptvec.pt.vpc_df:0 x0sug_fpc_lookup_vec.fplu_vec.lkup.dciptvec

211、.pt.vpc_df:0 x0module-1(DBG-elam-insel14)#ereport|grep df|grep vpcsug_lub_latch_results_vec.lub4_1.vpc_df:0 x1sug_fpx_lookup_vec.lkup.dciptvec.pt.vpc_df:0 x1sug_fpc_lookup_vec.fplu_vec.lkup.dciptvec.pt.vpc_df:0 x1sug_fpc_lookup_vec.fplu_vec.lkup.dciptvec.pt.vpc_df:0 x1DF LeafDF LeafNonNon-DF LeafDF

212、LeafBRKDCN-390097Debugging ACI Bridged Flows 2023 Cisco and/or its affiliates.All rights reserved.Cisco Public#CiscoLiveKnown Unicast Ingress LeafLeafLeafLeafLeafSpineSpineSpineSpineEP2EP2192.168.100.11/24192.168.100.11/240000.aaaa.bbbb0000.aaaa.bbbbLeafLeafLeaf looks at the dst mac to Leaf looks at

213、 the dst mac to determine if it should route or switchdetermine if it should route or switchBridge Domain Settings:Unicast Routing DisableUnknown Unicast FloodEP1EP1192.168.100.10/24192.168.100.10/240000.cccc.dddd0000.cccc.ddddPing 192.168.100.10leaf103#show endpoint mac 0000.cccc.dddd+-+-+-+-+VLAN/

214、Encap MAC Address InterfaceDomain VLAN IP Address +-+-+-+-+32/CL2022:vrf1 vxlan-14811121 0000.cccc.dddd tunnel1Lookup dst mac in ingress BDSince dst mac is not the Since dst mac is not the router(GW)mac,leaf does router(GW)mac,leaf does mac lookup in BD(usually mac lookup in BD(usually determined by

215、 ingress vlan)determined by ingress vlan)21leaf103#show int tun1Tunnel destination 10.0.176.67BRKDCN-390099 2023 Cisco and/or its affiliates.All rights reserved.Cisco Public#CiscoLiveKnown Unicast Ingress LeafLeafLeafLeafLeafSpineSpineSpineSpineEP2EP2192.168.100.11/24192.168.100.11/240000.aaaa.bbbb0

216、000.aaaa.bbbbLeafLeafBridge Domain Settings:Unicast Routing DisableUnknown Unicast FloodEP1EP1192.168.100.10/24192.168.100.10/240000.cccc.dddd0000.cccc.ddddPing 192.168.100.10vsh_lcdebug plat internal app elam asic 0trigger resettrigger init in-select 6 out-select 0set outer ipv4 src_ip 192.168.100.

217、11 dst_ip 192.168.100.10startELAMELAMBRKDCN-3900100 2023 Cisco and/or its affiliates.All rights reserved.Cisco Public#CiscoLiveKnown Unicast Ingress LeafForwarding VerificationsBridge Domain Settings:Unicast Routing DisableUnknown Unicast FloodOuter L2 Header-Destination MAC :0000.cccc.ddddSource MA

218、C :0000.aaaa.bbbbAccess Encap VLAN :3502(0 xDAE)Outer L3 Header-IP Protocol Number:ICMPDestination IP :192.168.100.10Source IP :192.168.100.11Other Forwarding Information-Encap Index is valid :yesEncap Index :34(0 x22)FINAL FORWARDING LOOKUP-Bits set in Final Forwarding Block:IFABRIC_IG UC TENANT MY

219、TEP BRIDGE HITLookup Drop-LU drop reason :no dropDest mac that is looked up within BDDest mac that is looked up within BDMake sure this is the expected vlanMake sure this is the expected vlanDest is tunnelDest is tunnelshow plat internal hal tunnel rtep apd=ifId IP RwEncapIdx=18010001 10.0.176.67 22

220、 Forward to this overlay TEPForward to this overlay TEPNot Dropped in lookups!Not Dropped in lookups!Unicast+Bridge(L2 lookup)+Destination KnownBRKDCN-3900101 2023 Cisco and/or its affiliates.All rights reserved.Cisco Public#CiscoLiveKnown Unicast Ingress LeafForwarding VerificationsBridge Domain Se

221、ttings:Unicast Routing DisableUnknown Unicast Floodereport|grep ovector ovector :152(0 x98)show platform internal hal l2 port gpd=IfId Ifname As AP Sl Sp Ss Ovec=1a01c000 Eth1/290 59 2 18 18 98Traffic is forwarded out Eth1/29!Traffic is forwarded out Eth1/29!BRKDCN-3900102 2023 Cisco and/or its affi

222、liates.All rights reserved.Cisco Public#CiscoLiveKnown Unicast Ingress LeafContract VerificationBridge Domain Settings:Unicast Routing DisableUnknown Unicast FloodContract Lookup Key-IP Protocol :ICMP(0 x1)L4 Src Port :2048(0 x800)L4 Dst Port :35914(0 x8C4A)sclass(src pcTag):49154(0 xC002)dclass(dst

223、 pcTag):49154(0 xC002)src pcTag is from local table :yesUnknown Unicast/Flood Packet:noContract Result-Contract Drop :noContract Applied :yesContract Hit :yesContract Aclqos Stats Index :131025(show sys int aclqos zoning-rules|grep-B 9 Idx:131025)Source and Dest EPG is the Source and Dest EPG is the

224、 same.Implicitly permit!same.Implicitly permit!(unless isolation enabled)(unless isolation enabled)Contract Applied and Contract Applied and no Drop!no Drop!BRKDCN-3900103 2023 Cisco and/or its affiliates.All rights reserved.Cisco Public#CiscoLiveKnown Unicast Egress LeafLeafLeafLeafLeafSpineSpineSp

225、ineSpineEP2EP2192.168.100.11/24192.168.100.11/240000.aaaa.bbbb0000.aaaa.bbbbLeafLeafPolicy was applied by ingress Policy was applied by ingress leaf.Dont apply contracts!leaf.Dont apply contracts!Bridge Domain Settings:Unicast Routing DisableUnknown Unicast FloodEP1EP1192.168.100.10/24192.168.100.10

226、/240000.cccc.dddd0000.cccc.ddddPing 192.168.100.10leaf101#show endpoint mac 0000.cccc.dddd+-+-+-+-+VLAN/Encap MAC Address InterfaceDomain VLAN IP Address +-+-+-+-+3/CL2022:vrf1 vlan-3501 0000.cccc.dddd po5Forward out portchannel 5 in vlan 3501!Since VNID is the BD VNID,forward Since VNID is the BD V

227、NID,forward based on dest endpoint macbased on dest endpoint mac34BRKDCN-3900104 2023 Cisco and/or its affiliates.All rights reserved.Cisco Public#CiscoLiveKnown Unicast Egress LeafLeafLeafLeafLeafSpineSpineSpineSpineEP2EP2192.168.100.11/24192.168.100.11/240000.aaaa.bbbb0000.aaaa.bbbbLeafLeafBridge

228、Domain Settings:Unicast Routing DisableUnknown Unicast FloodEP1EP1192.168.100.10/24192.168.100.10/240000.cccc.dddd0000.cccc.ddddPing 192.168.100.10debug plat internal tah elam asic 0trigger resettrigger init in-select 14 out-select 0set inner ipv4 src_ip 192.168.100.11 dst_ip 192.168.100.10startElam

229、ElamBRKDCN-3900105 2023 Cisco and/or its affiliates.All rights reserved.Cisco Public#CiscoLiveInner L2 Header-Inner Destination MAC :0000.cccc.ddddInner L3 Header-Destination IP :192.168.100.10Outer L4 Header-L4 Type :iVxLANSrc Policy Applied Bit:1Dst Policy Applied Bit:1VRF or BD VNID :14811121(0 x

230、E1FFF1)Sideband Information-ovector:146(0 x92)FINAL FORWARDING LOOKUP-Bits set in Final Forwarding Block:IFABRIC_EG UC INFRA ENCAP MYTEP BRIDGE HITLookup Drop-LU drop reason :no dropKnown Unicast Egress LeafBridge Domain Settings:Unicast Routing DisableUnknown Unicast Floodshow platform internal hal

231、 l2 port gpd=IfId Ifname As AP Sl Sp Ss Ovec=1a021000 Eth1/340 32 1 9 12 92Unicast+Bridge(L2 lookup)+Destination KnownContracts have already been Contracts have already been applied.No need to check.applied.No need to check.Forward out Eth1/34!Forward out Eth1/34!Mac lookup done in bridge Mac lookup

232、 done in bridge domain with this VNIDdomain with this VNIDBRKDCN-3900106Debugging ACI Routed Flows 2023 Cisco and/or its affiliates.All rights reserved.Cisco Public#CiscoLiveKnown Unicast Ingress LeafLeafLeafLeafLeafSpineSpineSpineSpineEP2EP2192.168.200.11/24192.168.200.11/240000.aaaa.bbbb0000.aaaa.

233、bbbbLeafLeafLeaf looks at the dst mac to Leaf looks at the dst mac to determine if it should route or switchdetermine if it should route or switchBridge Domain Settings:Unicast Routing EnabledEP1EP1192.168.100.10/24192.168.100.10/240000.cccc.dddd0000.cccc.ddddleaf103#show endpoint ip 192.168.100.10+

234、-+-+-+VLAN/MAC Address InterfaceDomain IP Address +-+-+-+CL2022:vrf1 192.168.100.10 tunnel1Lookup dst IP in ingress VRFSince dst mac is the router(GW)mac,leaf Since dst mac is the router(GW)mac,leaf does IP lookup in VRF of source IPdoes IP lookup in VRF of source IPleaf103#show int tun1Tunnel desti

235、nation 10.0.176.67Ping 192.168.100.1021BRKDCN-3900108 2023 Cisco and/or its affiliates.All rights reserved.Cisco Public#CiscoLiveKnown Unicast Ingress LeafLeafLeafLeafLeafSpineSpineSpineSpineEP2EP2192.168.200.11/24192.168.200.11/240000.aaaa.bbbb0000.aaaa.bbbbLeafLeafBridge Domain Settings:Unicast Ro

236、uting EnabledEP1EP1192.168.100.10/24192.168.100.10/240000.cccc.dddd0000.cccc.ddddPing 192.168.100.10Leaf forwards packet to remote Leaf forwards packet to remote TEP with VRF VNID setTEP with VRF VNID set4103#show sys internal epm endpoint ip 192.168.200.11!omittedBD vnid:16613259:VRF vnid:2523136sc

237、lass:32771103#show zoning-rule src-epg 32771 dst-epg 49154 scope 2523136+-+-+-+|RuleID|Name|Action|+-+-+-+|4209|CL2022:allow-all|permit|+-+-+-+103#show sys internal epm endpoint ip 192.168.100.10!omittedBD vnid:0:VRF vnid:2523136sclass:49154Leaf does contract Leaf does contract lookup based on src l

238、ookup based on src and dst pcTag valuesand dst pcTag values3Get SclassGet DclassCheck ContractBRKDCN-3900109 2023 Cisco and/or its affiliates.All rights reserved.Cisco Public#CiscoLiveKnown Unicast Ingress LeafLeafLeafLeafLeafSpineSpineSpineSpineEP2EP2192.168.200.11/24192.168.200.11/240000.aaaa.bbbb

239、0000.aaaa.bbbbLeafLeafEP1EP1192.168.100.10/24192.168.100.10/240000.cccc.dddd0000.cccc.ddddPing 192.168.100.10vsh_lcdebug plat internal app elam asic 0trigger resettrigger init in-select 6 out-select 0set outer ipv4 src_ip 192.168.200.11set outer ipv4 dst_ip 192.168.100.10startstatELAM STATUS=Asic 0

240、Slice 0 Status TriggeredAsic 0 Slice 1 Status ArmedELAMELAMBridge Domain Settings:Unicast Routing EnabledBRKDCN-3900110 2023 Cisco and/or its affiliates.All rights reserved.Cisco Public#CiscoLiveKnown Unicast Ingress LeafForwarding VerificationsOuter L2 Header-Destination MAC :0022.BDF8.19FFAccess E

241、ncap VLAN :3769(0 xEB9)Outer L3 Header-Destination IP :192.168.100.10Source IP :192.168.200.11Other Forwarding Information-Encap Index is valid :yesEncap Index :34(0 x22)FINAL FORWARDING LOOKUP-Bits set in Final Forwarding Block:IFABRIC_IG UC TENANT MYTEP ROUTE HITLookup Drop-LU drop reason :no drop

242、ACI Router Mac.Route this packet!ACI Router Mac.Route this packet!Make sure this is the expected vlanMake sure this is the expected vlanDest is tunnelDest is tunnelshow plat internal hal tunnel rtep apd=ifId IP RwEncapIdx=18010001 10.0.176.67 22 Forward to this overlay TEPForward to this overlay TEP

243、Not Dropped in lookups!Not Dropped in lookups!Unicast+Route(L3 lookup)+L3 Route FoundBridge Domain Settings:Unicast Routing EnabledBRKDCN-3900111 2023 Cisco and/or its affiliates.All rights reserved.Cisco Public#CiscoLiveKnown Unicast Ingress LeafForwarding Verificationsereport|grep ovector ovector

244、:152(0 x98)show platform internal hal l2 port gpd=IfId Ifname As AP Sl Sp Ss Ovec=1a01c000 Eth1/290 59 2 18 18 98Traffic is forwarded out Eth1/29!Traffic is forwarded out Eth1/29!Bridge Domain Settings:Unicast Routing EnabledBRKDCN-3900112 2023 Cisco and/or its affiliates.All rights reserved.Cisco P

245、ublic#CiscoLiveKnown Unicast Ingress LeafContract VerificationContract Lookup Key-IP Protocol :ICMP(0 x1)L4 Src Port :2048(0 x800)L4 Dst Port :31219(0 x79F3)sclass(src pcTag):32771(0 x8003)dclass(dst pcTag):49154(0 xC002)src pcTag is from local table :yesUnknown Unicast/Flood Packet:noContract Resul

246、t-Contract Drop :noContract Applied :yesContract Hit :yesContract Aclqos Stats Index :131025Source and Dest EPG Source and Dest EPG used for contract lookupused for contract lookupContract Applied and Contract Applied and no Drop!no Drop!Bridge Domain Settings:Unicast Routing EnabledBut how do I kno

247、w which But how do I know which contract this is actually hitting?contract this is actually hitting?BRKDCN-3900113 2023 Cisco and/or its affiliates.All rights reserved.Cisco Public#CiscoLiveKnown Unicast Ingress LeafContract VerificationContract Result-Contract Drop :noContract Applied :yesContract

248、Hit :yesContract Aclqos Stats Index:131025Hardware Index of Hardware Index of matching contractmatching contractBridge Domain Settings:Unicast Routing Enabledshow sys int aclqos zoning-rules|grep-B 9 Idx:130974=Rule ID:4163 Scope 8 Src EPG:32771 Dst EPG:49154 Filter 532Curr TCAM resource:=SDK Info=R

249、esult/Stats Idx:130974ZoningZoning-rule IDrule IDRun this from vsh_lcRun this from vsh_lcshow zoning-rule rule-id 4163+-+-+-+-+-+-+-+|Rule ID|SrcEPG|DstEPG|FilterID|Scope|Name|Action|+-+-+-+-+-+-+-+|4163|32771|49154|532|2523136|CL2022:allow-all|permit|+-+-+-+-+-+-+-+Traffic hit this contract!Traffic

250、 hit this contract!Run this from normal shellRun this from normal shellBRKDCN-3900114 2023 Cisco and/or its affiliates.All rights reserved.Cisco Public#CiscoLiveKnown Unicast Egress LeafLeafLeafLeafLeafSpineSpineSpineSpineEP2EP2192.168.200.11/24192.168.200.11/240000.aaaa.bbbb0000.aaaa.bbbbLeafLeafPo

251、licy was applied by ingress Policy was applied by ingress leaf.Dont apply contracts!leaf.Dont apply contracts!EP1EP1192.168.100.10/24192.168.100.10/240000.cccc.dddd0000.cccc.ddddleaf102#show endpoint ip 192.168.100.10+-+-+-+-+VLAN/Encap MAC Address InterfaceDomain VLAN IP Address +-+-+-+-+3 vlan-350

252、1 0000.cccc.dddd po5CL2022:vrf1 vlan-3501 192.168.100.10 po5Forward out portchannel 5 in vlan 3501!Since VNID is the VRF VNID,forward Since VNID is the VRF VNID,forward based on dest endpoint IPbased on dest endpoint IPBridge Domain Settings:Unicast Routing Enabled56Ping 192.168.100.10BRKDCN-3900115

253、 2023 Cisco and/or its affiliates.All rights reserved.Cisco Public#CiscoLiveKnown Unicast Egress LeafLeafLeafLeafLeafSpineSpineSpineSpineEP2EP2192.168.200.11/24192.168.200.11/240000.aaaa.bbbb0000.aaaa.bbbbLeafLeafEP1EP1192.168.100.10/24192.168.100.10/240000.cccc.dddd0000.cccc.ddddPing 192.168.100.10

254、debug plat internal tah elam asic 0trigger resettrigger init in-select 14 out-select 0set inner ipv4 src_ip 192.168.200.11 set inner ipv4 dst_ip 192.168.100.10startstatELAM STATUS=Asic 0 Slice 0 Status TriggeredAsic 0 Slice 1 Status ArmedElamElamBridge Domain Settings:Unicast Routing EnabledBRKDCN-3

255、900116 2023 Cisco and/or its affiliates.All rights reserved.Cisco Public#CiscoLiveInner L2 Header-Inner Destination MAC :000C.0C0C.0C0CInner L3 Header-Destination IP :192.168.100.10Outer L4 Header-L4 Type :iVxLANSrc Policy Applied Bit:1Dst Policy Applied Bit:1VRF or BD VNID :2523136(0 x268000)Sideba

256、nd Information-ovector:146(0 x92)FINAL FORWARDING LOOKUP-Bits set in Final Forwarding Block:IFABRIC_EG UC INFRA ENCAP MYTEP ROUTE HITLookup Drop-LU drop reason :no dropKnown Unicast Egress Leafshow platform internal hal l2 port gpd=IfId Ifname As AP Sl Sp Ss Ovec=1a021000 Eth1/340 32 1 9 12 92Unicas

257、t+Route(L3 lookup)+L3 Route FoundContracts have already been Contracts have already been applied.No need to check.applied.No need to check.Forward out Eth1/34!Forward out Eth1/34!IP lookup done in VRF with this VNIDIP lookup done in VRF with this VNIDBridge Domain Settings:Unicast Routing EnabledBRK

258、DCN-3900117 2023 Cisco and/or its affiliates.All rights reserved.Cisco Public#CiscoLiveProxied Unicast Ingress LeafLeafLeafLeafLeafSpineSpineSpineSpineEP2EP2192.168.200.11/24192.168.200.11/240000.aaaa.bbbb0000.aaaa.bbbbLeafLeafDst mac is router mac so Route!Dst mac is router mac so Route!Lookup dest

259、 IP in vrf of source IP.Lookup dest IP in vrf of source IP.Bridge Domain Settings:Unicast Routing EnabledEP1EP1192.168.100.10/24192.168.100.10/240000.cccc.dddd0000.cccc.ddddshow endpoint ip 192.168.100.10+-+-+-+VLAN/MAC Address InterfaceDomain IP Address +-+-+-+Lookup dst IP in ingress VRFPing 192.1

260、68.100.10show ip route 192.168.100.10 vrf CL2022:vrf1192.168.100.0/24,attached,direct,pervasive*via 10.0.176.66%overlay-1,1/0,staticrecursive next hop:10.0.176.66/32%overlay-1No endpoint learn,No endpoint learn,check route table!check route table!show isis dtep vrf overlay-1DTEP-Address Role Type10.

261、0.176.66 SPINE PHYSICAL,PROXY-ACAST-V4Send to Send to spine proxy!spine proxy!11a1b1cBRKDCN-3900118 2023 Cisco and/or its affiliates.All rights reserved.Cisco Public#CiscoLiveProxied Unicast Ingress LeafLeafLeafLeafLeafSpineSpineSpineSpineEP2EP2192.168.200.11/24192.168.200.11/240000.aaaa.bbbb0000.aa

262、aa.bbbbLeafLeafBridge Domain Settings:Unicast Routing EnabledEP1EP1192.168.100.10/24192.168.100.10/240000.cccc.dddd0000.cccc.ddddContract not applied for proxy lookups.Contract not applied for proxy lookups.Leaf forwards packet to spine Leaf forwards packet to spine proxy TEP with VRF VNID setproxy

263、TEP with VRF VNID set32Ping 192.168.100.10BRKDCN-3900119 2023 Cisco and/or its affiliates.All rights reserved.Cisco Public#CiscoLiveProxied Unicast Ingress LeafLeafLeafLeafLeafSpineSpineSpineSpineEP2EP2192.168.200.11/24192.168.200.11/240000.aaaa.bbbb0000.aaaa.bbbbLeafLeafEP1EP1192.168.100.10/24192.1

264、68.100.10/240000.cccc.dddd0000.cccc.ddddPing 192.168.100.10vsh_lcdebug plat internal app elam asic 0trigger resettrigger init in-select 6 out-select 0set outer ipv4 src_ip 192.168.200.11set outer ipv4 dst_ip 192.168.100.10startstatELAM STATUS=Asic 0 Slice 0 Status TriggeredAsic 0 Slice 1 Status Arme

265、dELAMELAMBridge Domain Settings:Unicast Routing EnabledBRKDCN-3900120 2023 Cisco and/or its affiliates.All rights reserved.Cisco Public#CiscoLiveProxied Unicast Ingress LeafForwarding Verifications121BRKDCN-3900Outer L2 Header-Destination MAC :0022.BDF8.19FFAccess Encap VLAN :3769(0 xEB9)Outer L3 He

266、ader-Destination IP :192.168.100.10Source IP :192.168.200.11Other Forwarding Information-Encap Index is valid :yesEncap Index :1(0 x1)FINAL FORWARDING LOOKUP-Bits set in Final Forwarding Block:IFABRIC_IG UC TENANT MYTEP ROUTE HITLookup Drop-LU drop reason :no dropACI Router Mac.Route this packet!ACI

267、 Router Mac.Route this packet!Make sure this is the expected vlanMake sure this is the expected vlanDest is tunnelDest is tunnelshow plat internal hal tunnel rtep apd=ifId IP RwEncapIdx=18010007 10.0.176.66 1 Forward to this overlay TEPForward to this overlay TEPNot Dropped in lookups!Not Dropped in

268、 lookups!Unicast+Route(L3 lookup)+L3 Route FoundBridge Domain Settings:Unicast Routing Enabled 2023 Cisco and/or its affiliates.All rights reserved.Cisco Public#CiscoLiveProxied Unicast Ingress LeafForwarding Verificationsereport|grep ovector ovector :152(0 x98)show platform internal hal l2 port gpd

269、=IfId Ifname As AP Sl Sp Ss Ovec=1a01c000 Eth1/290 59 2 18 18 98Traffic is forwarded out Eth1/29!Traffic is forwarded out Eth1/29!Bridge Domain Settings:Unicast Routing EnabledBRKDCN-3900122 2023 Cisco and/or its affiliates.All rights reserved.Cisco Public#CiscoLiveProxied Unicast Ingress LeafContra

270、ct VerificationContract Lookup Key-IP Protocol :ICMP(0 x1)L4 Src Port :2048(0 x800)L4 Dst Port :31219(0 x79F3)sclass(src pcTag):32771(0 x8003)dclass(dst pcTag):1(0 x1)src pcTag is from local table :yesUnknown Unicast/Flood Packet:noContract Result-Contract Drop :noContract Applied :noContract Hit :y

271、esContract Aclqos Stats Index :131025Dest EPG is 1 for fabric Dest EPG is 1 for fabric owned subnetsowned subnetsContract not applied Contract not applied since this is proxied!since this is proxied!Bridge Domain Settings:Unicast Routing EnabledBRKDCN-3900123 2023 Cisco and/or its affiliates.All rig

272、hts reserved.Cisco Public#CiscoLiveProxied Unicast SpineLeafLeafLeafLeafSpineSpineSpineSpineEP2EP2192.168.200.11/24192.168.200.11/240000.aaaa.bbbb0000.aaaa.bbbbLeafLeafSince this is proxied,Spine does Since this is proxied,Spine does COOP lookup.Since VRF vnid is set,COOP lookup.Since VRF vnid is se

273、t,Spine looks up IP rather than macSpine looks up IP rather than macBridge Domain Settings:Unicast Routing EnabledEP1EP1192.168.100.10/24192.168.100.10/240000.cccc.dddd0000.cccc.ddddPing 192.168.100.10spine#show coop internal info ip-db|grep-B 1-A 15 192.168.100.10-IP address:192.168.100.10Vrf:25231

274、36Num tunnels:1Tunnel address:10.0.176.67Tunnel ref count:1apic1#moquery-c ipv4Addr-f ipv4.Addr.addr=10.0.176.67*node-101/*dom-overlay-1/if-lo1/addr-10.0.176.67/32*node-102/*dom-overlay-1/if-lo1/addr-10.0.176.67/32Dest TEP of Leaf(s)that Dest TEP of Leaf(s)that own this Endpointown this Endpoint4BRK

275、DCN-3900124 2023 Cisco and/or its affiliates.All rights reserved.Cisco Public#CiscoLiveProxied Unicast Egress LeafLeafLeafLeafLeafSpineSpineSpineSpineEP2EP2192.168.200.11/24192.168.200.11/240000.aaaa.bbbb0000.aaaa.bbbbLeafLeafPolicy was NOT applied by Policy was NOT applied by ingress leaf.Apply con

276、tracts!ingress leaf.Apply contracts!EP1EP1192.168.100.10/24192.168.100.10/240000.cccc.dddd0000.cccc.ddddPing 192.168.100.10leaf102#show endpoint ip 192.168.100.10+-+-+-+-+VLAN/Encap MAC Address InterfaceDomain VLAN IP Address +-+-+-+-+3 vlan-3501 0000.cccc.dddd po5CL2022:vrf1 vlan-3501 192.168.100.1

277、0 po5Forward out portchannel 5 in vlan 3501!Since VNID is the VRF VNID,forward Since VNID is the VRF VNID,forward based on dest endpoint IPbased on dest endpoint IPBridge Domain Settings:Unicast Routing Enabled56BRKDCN-3900125 2023 Cisco and/or its affiliates.All rights reserved.Cisco Public#CiscoLi

278、veProxied Unicast Egress LeafLeafLeafLeafLeafSpineSpineSpineSpineEP2EP2192.168.200.11/24192.168.200.11/240000.aaaa.bbbb0000.aaaa.bbbbLeafLeafEP1EP1192.168.100.10/24192.168.100.10/240000.cccc.dddd0000.cccc.ddddPing 192.168.100.10debug plat internal tah elam asic 0trigger resettrigger init in-select 1

279、4 out-select 0set inner ipv4 src_ip 192.168.200.11 set inner ipv4 dst_ip 192.168.100.10startstatELAM STATUS=Asic 0 Slice 0 Status TriggeredAsic 0 Slice 1 Status ArmedElamElamBridge Domain Settings:Unicast Routing EnabledBRKDCN-3900126 2023 Cisco and/or its affiliates.All rights reserved.Cisco Public

280、#CiscoLiveInner L3 Header-Destination IP :192.168.100.10Outer L4 Header-L4 Type :iVxLANSrc Policy Applied Bit:0Dst Policy Applied Bit:0VRF or BD VNID :2523136(0 x268000)Sideband Information-ovector :146(0 x92)FINAL FORWARDING LOOKUP-Bits set in Final Forwarding Block:IFABRIC_EG UC INFRA ENCAP MYTEP

281、ROUTE HITLookup Drop-LU drop reason :no dropProxied Unicast Egress LeafForwarding Verificationsshow platform internal hal l2 port gpd=IfId Ifname As AP Sl Sp Ss Ovec=1a021000 Eth1/340 32 1 9 12 92Unicast+Route(L3 lookup)+L3 Route FoundContracts have not been applied yet!Contracts have not been appli

282、ed yet!Forward out Eth1/34!Forward out Eth1/34!IP lookup done in VRF with this VNIDIP lookup done in VRF with this VNIDBridge Domain Settings:Unicast Routing EnabledNot Dropped in lookups!Not Dropped in lookups!BRKDCN-3900127 2023 Cisco and/or its affiliates.All rights reserved.Cisco Public#CiscoLiv

283、eProxied Unicast Egress LeafContract VerificationContract Lookup Key-IP Protocol :ICMP(0 x1)L4 Src Port :2048(0 x800)L4 Dst Port :33226(0 x81CA)sclass(src pcTag):32771(0 x8003)dclass(dst pcTag):49154(0 xC002)src pcTag is from local table :noUnknown Unicast/Flood Packet:noContract Result-Contract Dro

284、p :noContract Applied :yesContract Hit :yesContract Aclqos Stats Index :131025Source and Dest EPG used Source and Dest EPG used for contract lookup.for contract lookup.Contract Applied and Contract Applied and no Drop!no Drop!Bridge Domain Settings:Unicast Routing EnabledBut how do I know which But

285、how do I know which contract this is actually hitting?contract this is actually hitting?BRKDCN-3900128 2023 Cisco and/or its affiliates.All rights reserved.Cisco Public#CiscoLiveProxied Unicast Egress LeafContract VerificationContract Result-Contract Drop :noContract Applied :yesContract Hit :yesCon

286、tract Aclqos Stats Index:81836Hardware Index of Hardware Index of matching contractmatching contractBridge Domain Settings:Unicast Routing Enabledshow sys int aclqos zoning-rules|grep-B 9 Idx:81836=Rule ID:4234 Scope 16 Src EPG:32771 Dst EPG:49154 Filter 532=SDK Info=Result/Stats Idx:81836ZoningZoni

287、ng-rule IDrule IDRun this from vsh_lcRun this from vsh_lcshow zoning-rule rule-id 4234+-+-+-+-+-+-+-+|Rule ID|SrcEPG|DstEPG|FilterID|Scope|Name|Action|+-+-+-+-+-+-+-+|4163|32771|49154|532|2523136|CL2022:allow-all|permit|+-+-+-+-+-+-+-+Traffic hit this contract!Traffic hit this contract!Run this from

288、 normal shellRun this from normal shellBRKDCN-3900129 2023 Cisco and/or its affiliates.All rights reserved.Cisco Public#CiscoLiveL3Out Destination Ingress LeafLeafLeafLeafLeafSpineSpineSpineSpineLeafLeafLeaf looks at the dst mac to Leaf looks at the dst mac to determine if it should route or switchd

289、etermine if it should route or switchBridge Domain Settings:Unicast Routing EnabledEP1EP1192.168.100.10/24192.168.100.10/240000.cccc.dddd0000.cccc.ddddSince dst mac is the router(GW)mac,Since dst mac is the router(GW)mac,leaf does IP lookup in VRF of source IPleaf does IP lookup in VRF of source IPP

290、ing 10.99.99.100External IPExternal IP10.99.99.10010.99.99.100show endpoint ip 10.99.99.100+-+-+-+VLAN/MAC Address InterfaceDomain IP Address +-+-+-+Lookup dst IP in ingress VRFshow ip route 10.99.99.100 vrf CL2022:vrf110.99.99.0/24,ubest/mbest:1/0*via 10.0.64.70%overlay-1,200/20,bgp-65100recursive

291、next hop:10.0.64.70/32%overlay-1No endpoint learn,No endpoint learn,check route table!check route table!acidiag fnvread|grep 10.0.64.70Name IP Address Role-leaf10310.0.64.70/32 leafSend to BL Send to BL PTEP!PTEP!122a2b2cBRKDCN-3900130 2023 Cisco and/or its affiliates.All rights reserved.Cisco Publi

292、c#CiscoLiveL3Out Destination Ingress LeafLeafLeafLeafLeafSpineSpineSpineSpineLeafLeafBridge Domain Settings:Unicast Routing EnabledEP1EP1192.168.100.10/24192.168.100.10/240000.cccc.dddd0000.cccc.ddddLeaf derives dest pcTag for Leaf derives dest pcTag for contract lookup based on l3out contract looku

293、p based on l3out policy prefixpolicy prefixPing 10.99.99.100External IPExternal IP10.99.99.10010.99.99.100leaf#vsh_lc vsh_lc-c show forwarding route 10.99.99.100 platf vrf CL2022:vrf1c show forwarding route 10.99.99.100 platf vrf CL2022:vrf1!Policy Prefix 10.99.99.0/24Policy Prefix 10.99.99.0/24!vrf

294、:16(0 x10),routed_if:0 x0 epc_classepc_class:3277232772(0 x8004)3Leaf forwards packet to Leaf forwards packet to remote TEP with VRF VNID setremote TEP with VRF VNID set4BRKDCN-3900131 2023 Cisco and/or its affiliates.All rights reserved.Cisco Public#CiscoLiveL3Out Destination Ingress LeafLeafLeafLe

295、afLeafSpineSpineSpineSpineLeafLeafBridge Domain Settings:Unicast Routing EnabledEP1EP1192.168.100.10/24192.168.100.10/240000.cccc.dddd0000.cccc.ddddPing 10.99.99.100External IPExternal IP10.99.99.10010.99.99.100vsh_lcdebug plat internal tah elam asic 0trigger resettrigger init in-select 6 out-select

296、 0set outer ipv4 src_ip 192.168.100.10set outer ipv4 dst_ip 10.99.99.100startstatELAM STATUS=Asic 0 Slice 0 Status TriggeredAsic 0 Slice 1 Status ArmedELAMELAMBRKDCN-3900132 2023 Cisco and/or its affiliates.All rights reserved.Cisco Public#CiscoLiveL3Out Destination Ingress LeafForwarding Verificati

297、onsOuter L2 Header-Destination MAC :0022.BDF8.19FFAccess Encap VLAN :3501(0 xDAD)Outer L3 Header-Destination IP :10.99.99.100Source IP :192.168.100.10Other Forwarding Information-Encap Index is valid :yesEncap Index :37(0 x25)FINAL FORWARDING LOOKUP-Bits set in Final Forwarding Block:IFABRIC_IG UC T

298、ENANT MYTEP ROUTE HITLookup Drop-LU drop reason :no dropACI Router Mac.Route this packet!ACI Router Mac.Route this packet!Make sure this is the expected vlanMake sure this is the expected vlanDest is tunnelDest is tunnelshow plat internal hal tunnel rtep apd=ifId IP RwEncapIdx=18010004 10.0.64.70 25

299、 Forward to this overlay TEPForward to this overlay TEPNot Dropped in lookups!Not Dropped in lookups!Unicast+Route(L3 lookup)+L3 Route FoundBridge Domain Settings:Unicast Routing EnabledBRKDCN-3900133 2023 Cisco and/or its affiliates.All rights reserved.Cisco Public#CiscoLiveL3Out Destination Ingres

300、s LeafForwarding Verificationsereport|grep ovector ovector :48(0 x30)show platform internal hal l2 port gpd=IfId Ifname As AP Sl Sp Ss Ovec=1a035000 Eth1/540 19 0 18 30 30Traffic is forwarded out Eth1/54!Traffic is forwarded out Eth1/54!Bridge Domain Settings:Unicast Routing EnabledBRKDCN-3900134 20

301、23 Cisco and/or its affiliates.All rights reserved.Cisco Public#CiscoLiveL3Out Destination Ingress LeafContract VerificationContract Lookup Key-IP Protocol :ICMP(0 x1)L4 Src Port :2048(0 x800)L4 Dst Port :12063(0 x2F1F)sclass(src pcTag):49154(0 xC002)dclass(dst pcTag):32772(0 x8004)src pcTag is from

302、 local table :yesUnknown Unicast/Flood Packet:noContract Result-Contract Drop :noContract Applied :yesContract Hit :yesContract Aclqos Stats Index :81765Source and Dest EPG Source and Dest EPG used for contract lookupused for contract lookupContract Applied and Contract Applied and no Drop!no Drop!B

303、ridge Domain Settings:Unicast Routing EnabledBut how do I know which But how do I know which contract this is actually hitting?contract this is actually hitting?BRKDCN-3900135 2023 Cisco and/or its affiliates.All rights reserved.Cisco Public#CiscoLiveL3Out Destination Ingress LeafContract Verificati

304、onContract Result-Contract Drop :noContract Applied :yesContract Hit :yesContract Aclqos Stats Index:81765Hardware Index of Hardware Index of matching contractmatching contractBridge Domain Settings:Unicast Routing Enabledshow sys int aclqos zoning-rules|grep-B 9 Idx:81765=Rule ID:4248 Scope 16 Src

305、EPG:0 Dst EPG:32772 Filter 532Curr TCAM resource:=SDK Info=Result/Stats Idx:81765ZoningZoning-rule IDrule IDRun this from vsh_lcRun this from vsh_lcshow zoning-rule rule-id 4248+-+-+-+-+-+-+-+|Rule ID|SrcEPG|DstEPG|FilterID|Scope|Name|Action|+-+-+-+-+-+-+-+|4248|0|32772|532|2523136|CL2022:l3out-allo

306、w-all|permit|+-+-+-+-+-+-+-+Traffic hit this contract!Traffic hit this contract!Run this from normal shellRun this from normal shellBRKDCN-3900136 2023 Cisco and/or its affiliates.All rights reserved.Cisco Public#CiscoLiveshow mac address addr 0005.73ff.593c vl 25VLAN MAC Address Ports-+-+-*25 0005.

307、73ff.593c eth1/27/4L3Out Destination Egress LeafLeafLeafLeafLeafSpineSpineSpineSpineLeafLeafBridge Domain Settings:Unicast Routing EnabledEP1EP1192.168.100.10/24192.168.100.10/240000.cccc.dddd0000.cccc.ddddSince received VNID is the VRF VNID,Since received VNID is the VRF VNID,forward based on dest

308、endpoint IPforward based on dest endpoint IPPing 10.99.99.100External IPExternal IP10.99.99.10010.99.99.100show endpoint ip 10.99.99.100+-+-+-+VLAN/MAC Address InterfaceDomain IP Address +-+-+-+Lookup dst IP in received VRFshow ip route 10.99.99.100 vrf CL2022:vrf110.99.99.0/24,ubest/mbest:1/0*via 1

309、0.55.0.100,vlan25,110/20,ospf,type-2No endpoint learn,No endpoint learn,check route table!check route table!show ip arp 10.55.0.100 vrf CL2022:vrf1Address MAC Address Interface10.55.0.100 0005.73ff.593c vlan25Forward based on ARP Forward based on ARP and MAC Adajcenciesand MAC Adajcencies55a5b5c5dPo

310、licy was applied by ingress Policy was applied by ingress leaf.No need to apply contractsleaf.No need to apply contracts6BRKDCN-3900137 2023 Cisco and/or its affiliates.All rights reserved.Cisco Public#CiscoLiveL3Out Destination Egress LeafLeafLeafLeafLeafSpineSpineSpineSpineLeafLeafBridge Domain Se

311、ttings:Unicast Routing EnabledEP1EP1192.168.100.10/24192.168.100.10/240000.cccc.dddd0000.cccc.ddddPing 10.99.99.100External IPExternal IP10.99.99.10010.99.99.100debug plat internal app elam asic 0trigger resettrigger init in-select 14 out-select 0set inner ipv4 src_ip 192.168.100.10 set inner ipv4 d

312、st_ip 10.99.99.100startstatELAM STATUS=Asic 0 Slice 0 Status TriggeredAsic 0 Slice 1 Status ArmedElamElamBRKDCN-3900138 2023 Cisco and/or its affiliates.All rights reserved.Cisco Public#CiscoLiveL3Out Destination Egress LeafInner L2 Header-Inner Destination MAC :000C.0C0C.0C0CInner L3 Header-Destina

313、tion IP :10.99.99.100Outer L4 Header-L4 Type :iVxLANSrc Policy Applied Bit:1Dst Policy Applied Bit:1VRF or BD VNID :2523136(0 x268000)Sideband Information-ovector:147(0 x93)FINAL FORWARDING LOOKUP-Bits set in Final Forwarding Block:IFABRIC_EG UC INFRA ENCAP MYTEP ROUTE HITLookup Drop-LU drop reason

314、:no dropshow platform internal hal l2 port gpd=IfId Ifname As AP Sl Sp Ss Ovec=4301a000 Eth1/27/40 54 2 13 13 93Unicast+Route(L3 lookup)+L3 Route FoundContracts have already been Contracts have already been applied.No need to check.applied.No need to check.Forward out Eth1/27/4!Forward out Eth1/27/4

315、!IP lookup done in VRF with this VNIDIP lookup done in VRF with this VNIDBridge Domain Settings:Unicast Routing EnabledBRKDCN-3900139 2023 Cisco and/or its affiliates.All rights reserved.Cisco Public#CiscoLiveL3Out Source Ingress Border LeafLeafLeafLeafLeafSpineSpineSpineSpineLeafLeafBridge Domain S

316、ettings:Unicast Routing EnabledEP1EP1192.168.100.10/24192.168.100.10/240000.cccc.dddd0000.cccc.ddddICMP ReplyExternal IPExternal IP10.99.99.10010.99.99.100If VRF is in ingress mode,BL If VRF is in ingress mode,BL doesnt apply policydoesnt apply policyForward based on longest prefixForward based on l

317、ongest prefix-match within match within source VRF.EP learns are always longest.source VRF.EP learns are always longest.If dest IP is not learned endpoint and If dest IP is not learned endpoint and subnet is BD subnet,proxy!subnet is BD subnet,proxy!Refer back to the Routed Known Unicast and Proxied

318、 Unicast for more verifications12a2bBRKDCN-3900140 2023 Cisco and/or its affiliates.All rights reserved.Cisco Public#CiscoLiveFill out your session surveys!Attendees who fill out a minimum of four session surveys and the overall event survey will get Cisco Live-branded socks(while supplies last)!The

319、se points help you get on the leaderboard and increase your chances of winning daily and grand prizesAttendees will also earn 100 points in the Cisco Live Challenge for every survey completed.BRKDCN-3900141 2023 Cisco and/or its affiliates.All rights reserved.Cisco PublicContinue your educationVisit the Cisco Showcase for related demosBook your one-on-oneMeet the Engineer meetingAttend the interactive education with DevNet,Capture the Flag,and Walk-in LabsVisit the On-Demand Library for more sessions at www.CiscoL you#CiscoLive#CiscoLive