Re: [c-nsp] Nexus Architecture question
On Tue, 6 Jul 2021 at 21:37, Drew Weaver wrote: > Not sure how that would've gotten through QA but I am glad it was finally > fixed in newer hardware. You assume there was any QA. Cheers, James. ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/
Re: [c-nsp] Nexus Architecture question
Hello, Just to circle back since there appears to be some controversy regarding whether or not the original issue was being caused by a problem where NXOS doesn't compensate for a flaw in the hardware. I have confirmed that the same config on the same version works fine on the 93108TC-EX so it does appear that this is a hardware flaw that the OS is not compensating for on the Broadcom based platforms. Not sure how that would've gotten through QA but I am glad it was finally fixed in newer hardware. Thanks, -Drew From: Tim Stevenson (tstevens) Sent: Friday, June 11, 2021 11:55 AM To: Drew Weaver ; 'Jeffrey G. Fitzwater' ; 'cisco-nsp@puck.nether.net' Subject: RE: Nexus Architecture question I don't know that we can/should draw that conclusion - as you mentioned, you opened a TAC case but from my understanding it was never driven to a terminal resolution - either "known limitation, live with it" or "bug, we will/won't fix it" or "you're doing it wrong". I tested this on 2nd gen n9k and it works fine, I don't have 1st gen gear in my lab so can't empirically confirm that the broadcom-based line cards do or don't behave properly and why, or whether there's an alternative config that does work. One wrinkle here is that these linecards are now well past end of sale, and in fact are past End of Vulnerability/Security Support as well, so maybe it's a moot point anyway. For 10GBASE-T in the 2nd gen I'd suggest the X9788TC-FX linecard and FM-E2 fabric module, these should behave properly with the config I posted earlier. Tim From: Drew Weaver mailto:drew.wea...@thenap.com>> Sent: Friday, June 11, 2021 8:20 AM To: 'Jeffrey G. Fitzwater' mailto:jf...@princeton.edu>>; Tim Stevenson (tstevens) mailto:tstev...@cisco.com>>; 'cisco-nsp@puck.nether.net' mailto:cisco-nsp@puck.nether.net>> Subject: RE: Nexus Architecture question Yes, the problem is that the OS doesn't compensate for the flaws in the Broadcom based line cards. Thanks, -Drew From: Jeffrey G. Fitzwater mailto:jf...@princeton.edu>> Sent: Friday, June 11, 2021 11:19 AM To: Drew Weaver mailto:drew.wea...@thenap.com>>; 'Tim Stevenson (tstevens)' mailto:tstev...@cisco.com>>; 'cisco-nsp@puck.nether.net' mailto:cisco-nsp@puck.nether.net>> Subject: Re: Nexus Architecture question I am not sure this question was asked in this thread, but are you using a custom COPP and not the default? If you have a custom COPP you must apply the new policy with that name prefix i.e. router-core-copp-acl-hsrp Vs copp-acl-hsrp. We do this on our 7 and 9ks so that any new code does not override the custom COPP policies. Jeff Fitzwater Princeton University From: cisco-nsp mailto:cisco-nsp-boun...@puck.nether.net>> on behalf of Drew Weaver mailto:drew.wea...@thenap.com>> Date: Friday, June 11, 2021 at 08:43 To: Drew Weaver mailto:drew.wea...@thenap.com>>, 'Tim Stevenson (tstevens)' mailto:tstev...@cisco.com>>, 'cisco-nsp@puck.nether.net' mailto:cisco-nsp@puck.nether.net>> Subject: Re: [c-nsp] Nexus Architecture question Just for the list this ended up being a Nexus/NXOS limitation. -Original Message- From: cisco-nsp mailto:cisco-nsp-boun...@puck.nether.net>> On Behalf Of Drew Weaver Sent: Tuesday, June 8, 2021 11:07 AM To: 'Tim Stevenson (tstevens)' mailto:tstev...@cisco.com>>; 'cisco-nsp@puck.nether.net' mailto:cisco-nsp@puck.nether.net>> Subject: Re: [c-nsp] Nexus Architecture question Sure, N9K-SUP-B version 9.3(6) Also tried: 7.0(3)I7(8) Thanks, -Drew -Original Message- From: Tim Stevenson (tstevens) mailto:tstev...@cisco.com>> Sent: Friday, June 4, 2021 4:40 PM To: Drew Weaver mailto:drew.wea...@thenap.com>>; 'cisco-nsp@puck.nether.net' mailto:cisco-nsp@puck.nether.net>> Subject: RE: Nexus Architecture question Hi Drew, Can you specify hardware platform and software version here? I am not seeing what you're seeing, the config I sent blocks a BGP port scan in nmap, and prevents BGP peering to anything other than the specified IP. I am testing on Nexus 9500 with 10.1 NXOS; I suspect you are using some other platform eg n5k or n7k? I may be able to try it out here depending on what you're using. Thanks, Tim -Original Message- From: Drew Weaver mailto:drew.wea...@thenap.com>> Sent: Thursday, June 3, 2021 12:37 PM To: Tim Stevenson (tstevens) mailto:tstev...@cisco.com>>; 'cisco-nsp@puck.nether.net' mailto:cisco-nsp@puck.nether.net>> Subject: RE: Nexus Architecture question Hi Tim, I replied off-list with additional details but I wanted to reply on list to answer your specific questions: I copied the strict policy and then edited it with
Re: [c-nsp] Nexus Architecture question
--- Begin Message --- I don't know that we can/should draw that conclusion - as you mentioned, you opened a TAC case but from my understanding it was never driven to a terminal resolution - either "known limitation, live with it" or "bug, we will/won't fix it" or "you're doing it wrong". I tested this on 2nd gen n9k and it works fine, I don't have 1st gen gear in my lab so can't empirically confirm that the broadcom-based line cards do or don't behave properly and why, or whether there's an alternative config that does work. One wrinkle here is that these linecards are now well past end of sale, and in fact are past End of Vulnerability/Security Support as well, so maybe it's a moot point anyway. For 10GBASE-T in the 2nd gen I'd suggest the X9788TC-FX linecard and FM-E2 fabric module, these should behave properly with the config I posted earlier. Tim From: Drew Weaver Sent: Friday, June 11, 2021 8:20 AM To: 'Jeffrey G. Fitzwater' ; Tim Stevenson (tstevens) ; 'cisco-nsp@puck.nether.net' Subject: RE: Nexus Architecture question Yes, the problem is that the OS doesn't compensate for the flaws in the Broadcom based line cards. Thanks, -Drew From: Jeffrey G. Fitzwater mailto:jf...@princeton.edu>> Sent: Friday, June 11, 2021 11:19 AM To: Drew Weaver mailto:drew.wea...@thenap.com>>; 'Tim Stevenson (tstevens)' mailto:tstev...@cisco.com>>; 'cisco-nsp@puck.nether.net' mailto:cisco-nsp@puck.nether.net>> Subject: Re: Nexus Architecture question I am not sure this question was asked in this thread, but are you using a custom COPP and not the default? If you have a custom COPP you must apply the new policy with that name prefix i.e. router-core-copp-acl-hsrp Vs copp-acl-hsrp. We do this on our 7 and 9ks so that any new code does not override the custom COPP policies. Jeff Fitzwater Princeton University From: cisco-nsp mailto:cisco-nsp-boun...@puck.nether.net>> on behalf of Drew Weaver mailto:drew.wea...@thenap.com>> Date: Friday, June 11, 2021 at 08:43 To: Drew Weaver mailto:drew.wea...@thenap.com>>, 'Tim Stevenson (tstevens)' mailto:tstev...@cisco.com>>, 'cisco-nsp@puck.nether.net' mailto:cisco-nsp@puck.nether.net>> Subject: Re: [c-nsp] Nexus Architecture question Just for the list this ended up being a Nexus/NXOS limitation. -Original Message- From: cisco-nsp mailto:cisco-nsp-boun...@puck.nether.net>> On Behalf Of Drew Weaver Sent: Tuesday, June 8, 2021 11:07 AM To: 'Tim Stevenson (tstevens)' mailto:tstev...@cisco.com>>; 'cisco-nsp@puck.nether.net' mailto:cisco-nsp@puck.nether.net>> Subject: Re: [c-nsp] Nexus Architecture question Sure, N9K-SUP-B version 9.3(6) Also tried: 7.0(3)I7(8) Thanks, -Drew -Original Message- From: Tim Stevenson (tstevens) mailto:tstev...@cisco.com>> Sent: Friday, June 4, 2021 4:40 PM To: Drew Weaver mailto:drew.wea...@thenap.com>>; 'cisco-nsp@puck.nether.net' mailto:cisco-nsp@puck.nether.net>> Subject: RE: Nexus Architecture question Hi Drew, Can you specify hardware platform and software version here? I am not seeing what you're seeing, the config I sent blocks a BGP port scan in nmap, and prevents BGP peering to anything other than the specified IP. I am testing on Nexus 9500 with 10.1 NXOS; I suspect you are using some other platform eg n5k or n7k? I may be able to try it out here depending on what you're using. Thanks, Tim -Original Message- From: Drew Weaver mailto:drew.wea...@thenap.com>> Sent: Thursday, June 3, 2021 12:37 PM To: Tim Stevenson (tstevens) mailto:tstev...@cisco.com>>; 'cisco-nsp@puck.nether.net' mailto:cisco-nsp@puck.nether.net>> Subject: RE: Nexus Architecture question Hi Tim, I replied off-list with additional details but I wanted to reply on list to answer your specific questions: I copied the strict policy and then edited it with these classes at the top: policy-map type control-plane Test6 class custom-copp-system-p-bgp-allow police cir 19000 pps bc 128 packets conform transmit violate drop class custom-copp-system-p-bgp-deny police cir 0 pps bc 1 packets conform transmit violate drop These are the configured class-maps: Class-map type control-plane match-any custom-copp-system-p-bgp-allow match access-group name custom-copp-system-p-acl-bgp-allow Class-map type control-plane match-any custom-copp-system-p-bgp-deny match access-group name custom-copp-system-p-acl-bgp-deny These are the configured ACLs: IP access list custom-copp-system-p-acl-bgp-allow 3 permit tcp 192.168.1.2/32 gt 1023 any eq bgp 4 permit tcp 192.168.1.2/32 eq bgp any gt 1023 IP access list custom-copp-system-p-acl-bgp-deny 1 permit tcp any any eq bgp
Re: [c-nsp] Nexus Architecture question
Yes, the problem is that the OS doesn't compensate for the flaws in the Broadcom based line cards. Thanks, -Drew From: Jeffrey G. Fitzwater Sent: Friday, June 11, 2021 11:19 AM To: Drew Weaver ; 'Tim Stevenson (tstevens)' ; 'cisco-nsp@puck.nether.net' Subject: Re: Nexus Architecture question I am not sure this question was asked in this thread, but are you using a custom COPP and not the default? If you have a custom COPP you must apply the new policy with that name prefix i.e. router-core-copp-acl-hsrp Vs copp-acl-hsrp. We do this on our 7 and 9ks so that any new code does not override the custom COPP policies. Jeff Fitzwater Princeton University From: cisco-nsp mailto:cisco-nsp-boun...@puck.nether.net>> on behalf of Drew Weaver mailto:drew.wea...@thenap.com>> Date: Friday, June 11, 2021 at 08:43 To: Drew Weaver mailto:drew.wea...@thenap.com>>, 'Tim Stevenson (tstevens)' mailto:tstev...@cisco.com>>, 'cisco-nsp@puck.nether.net' mailto:cisco-nsp@puck.nether.net>> Subject: Re: [c-nsp] Nexus Architecture question Just for the list this ended up being a Nexus/NXOS limitation. -Original Message- From: cisco-nsp mailto:cisco-nsp-boun...@puck.nether.net>> On Behalf Of Drew Weaver Sent: Tuesday, June 8, 2021 11:07 AM To: 'Tim Stevenson (tstevens)' mailto:tstev...@cisco.com>>; 'cisco-nsp@puck.nether.net' mailto:cisco-nsp@puck.nether.net>> Subject: Re: [c-nsp] Nexus Architecture question Sure, N9K-SUP-B version 9.3(6) Also tried: 7.0(3)I7(8) Thanks, -Drew -Original Message- From: Tim Stevenson (tstevens) mailto:tstev...@cisco.com>> Sent: Friday, June 4, 2021 4:40 PM To: Drew Weaver mailto:drew.wea...@thenap.com>>; 'cisco-nsp@puck.nether.net' mailto:cisco-nsp@puck.nether.net>> Subject: RE: Nexus Architecture question Hi Drew, Can you specify hardware platform and software version here? I am not seeing what you're seeing, the config I sent blocks a BGP port scan in nmap, and prevents BGP peering to anything other than the specified IP. I am testing on Nexus 9500 with 10.1 NXOS; I suspect you are using some other platform eg n5k or n7k? I may be able to try it out here depending on what you're using. Thanks, Tim -Original Message- From: Drew Weaver mailto:drew.wea...@thenap.com>> Sent: Thursday, June 3, 2021 12:37 PM To: Tim Stevenson (tstevens) mailto:tstev...@cisco.com>>; 'cisco-nsp@puck.nether.net' mailto:cisco-nsp@puck.nether.net>> Subject: RE: Nexus Architecture question Hi Tim, I replied off-list with additional details but I wanted to reply on list to answer your specific questions: I copied the strict policy and then edited it with these classes at the top: policy-map type control-plane Test6 class custom-copp-system-p-bgp-allow police cir 19000 pps bc 128 packets conform transmit violate drop class custom-copp-system-p-bgp-deny police cir 0 pps bc 1 packets conform transmit violate drop These are the configured class-maps: Class-map type control-plane match-any custom-copp-system-p-bgp-allow match access-group name custom-copp-system-p-acl-bgp-allow Class-map type control-plane match-any custom-copp-system-p-bgp-deny match access-group name custom-copp-system-p-acl-bgp-deny These are the configured ACLs: IP access list custom-copp-system-p-acl-bgp-allow 3 permit tcp 192.168.1.2/32 gt 1023 any eq bgp 4 permit tcp 192.168.1.2/32 eq bgp any gt 1023 IP access list custom-copp-system-p-acl-bgp-deny 1 permit tcp any any eq bgp 10 permit tcp any gt 1023 any eq bgp 20 permit tcp any eq bgp any gt 1023 control-plane service-policy input Test6 The primary difference that I notice between the two configurations is that your allow is set to cos 7 and on mine it is not specified and yours is policing on bps and mine is policing on pps. (I am pretty certain that throughout the process I have tried it both ways). I find it a bit hard to follow that on this platform there is no way to write "just block everything that matches this class" you still have to give it a "burst" [which I obviously don't want] I understand that behind the scenes the control plane is supposed to "do the right thing" but it is just difficult to understand that from reading it. With this configuration applied: 1) Any host that NMAPs the device sees 179 open. 2) BGP sessions establish (if configured on the Nexus) for hosts that are not 192.168.1.2/32 Is there any way to find out what class traffic from a specific SRC IP or to a specific port is getting caught at in a CoPP policy? I suspect that the TCP/179 traffic is just bypassing these classes altogether and most likely are getting handled further down in the CoPP policy but I haven't been able to prove that. Thanks
Re: [c-nsp] Nexus Architecture question
I am not sure this question was asked in this thread, but are you using a custom COPP and not the default? If you have a custom COPP you must apply the new policy with that name prefix i.e. router-core-copp-acl-hsrp Vs copp-acl-hsrp. We do this on our 7 and 9ks so that any new code does not override the custom COPP policies. Jeff Fitzwater Princeton University From: cisco-nsp on behalf of Drew Weaver Date: Friday, June 11, 2021 at 08:43 To: Drew Weaver , 'Tim Stevenson (tstevens)' , 'cisco-nsp@puck.nether.net' Subject: Re: [c-nsp] Nexus Architecture question Just for the list this ended up being a Nexus/NXOS limitation. -Original Message- From: cisco-nsp On Behalf Of Drew Weaver Sent: Tuesday, June 8, 2021 11:07 AM To: 'Tim Stevenson (tstevens)' ; 'cisco-nsp@puck.nether.net' Subject: Re: [c-nsp] Nexus Architecture question Sure, N9K-SUP-B version 9.3(6) Also tried: 7.0(3)I7(8) Thanks, -Drew -Original Message- From: Tim Stevenson (tstevens) Sent: Friday, June 4, 2021 4:40 PM To: Drew Weaver ; 'cisco-nsp@puck.nether.net' Subject: RE: Nexus Architecture question Hi Drew, Can you specify hardware platform and software version here? I am not seeing what you're seeing, the config I sent blocks a BGP port scan in nmap, and prevents BGP peering to anything other than the specified IP. I am testing on Nexus 9500 with 10.1 NXOS; I suspect you are using some other platform eg n5k or n7k? I may be able to try it out here depending on what you're using. Thanks, Tim -Original Message- From: Drew Weaver Sent: Thursday, June 3, 2021 12:37 PM To: Tim Stevenson (tstevens) ; 'cisco-nsp@puck.nether.net' Subject: RE: Nexus Architecture question Hi Tim, I replied off-list with additional details but I wanted to reply on list to answer your specific questions: I copied the strict policy and then edited it with these classes at the top: policy-map type control-plane Test6 class custom-copp-system-p-bgp-allow police cir 19000 pps bc 128 packets conform transmit violate drop class custom-copp-system-p-bgp-deny police cir 0 pps bc 1 packets conform transmit violate drop These are the configured class-maps: Class-map type control-plane match-any custom-copp-system-p-bgp-allow match access-group name custom-copp-system-p-acl-bgp-allow Class-map type control-plane match-any custom-copp-system-p-bgp-deny match access-group name custom-copp-system-p-acl-bgp-deny These are the configured ACLs: IP access list custom-copp-system-p-acl-bgp-allow 3 permit tcp 192.168.1.2/32 gt 1023 any eq bgp 4 permit tcp 192.168.1.2/32 eq bgp any gt 1023 IP access list custom-copp-system-p-acl-bgp-deny 1 permit tcp any any eq bgp 10 permit tcp any gt 1023 any eq bgp 20 permit tcp any eq bgp any gt 1023 control-plane service-policy input Test6 The primary difference that I notice between the two configurations is that your allow is set to cos 7 and on mine it is not specified and yours is policing on bps and mine is policing on pps. (I am pretty certain that throughout the process I have tried it both ways). I find it a bit hard to follow that on this platform there is no way to write "just block everything that matches this class" you still have to give it a "burst" [which I obviously don't want] I understand that behind the scenes the control plane is supposed to "do the right thing" but it is just difficult to understand that from reading it. With this configuration applied: 1) Any host that NMAPs the device sees 179 open. 2) BGP sessions establish (if configured on the Nexus) for hosts that are not 192.168.1.2/32 Is there any way to find out what class traffic from a specific SRC IP or to a specific port is getting caught at in a CoPP policy? I suspect that the TCP/179 traffic is just bypassing these classes altogether and most likely are getting handled further down in the CoPP policy but I haven't been able to prove that. Thanks so much. -Drew -Original Message- From: Tim Stevenson (tstevens) Sent: Wednesday, June 2, 2021 5:31 PM To: Drew Weaver ; 'cisco-nsp@puck.nether.net' Subject: RE: Nexus Architecture question Hi Drew, In answer to your question about BGP, the BGP process runs only on the supervisor engine, it does not run on the linecards or anywhere else. It's a single process, not a per-interface process or anything like that. Curious how exactly you are configuring CoPP to filter this? With default CoPP, I get an "open/tcpwrapped" (green) response on TCP 179; I was able to get a "filtered" (red) response by adding a CoPP class that matches on BGP and polices to a CIR of 0. I preceeded that class with a class that matches on a specific neighborship - that BGP neighborship is successfully established while nmap still
Re: [c-nsp] Nexus Architecture question
Just for the list this ended up being a Nexus/NXOS limitation. -Original Message- From: cisco-nsp On Behalf Of Drew Weaver Sent: Tuesday, June 8, 2021 11:07 AM To: 'Tim Stevenson (tstevens)' ; 'cisco-nsp@puck.nether.net' Subject: Re: [c-nsp] Nexus Architecture question Sure, N9K-SUP-B version 9.3(6) Also tried: 7.0(3)I7(8) Thanks, -Drew -Original Message- From: Tim Stevenson (tstevens) Sent: Friday, June 4, 2021 4:40 PM To: Drew Weaver ; 'cisco-nsp@puck.nether.net' Subject: RE: Nexus Architecture question Hi Drew, Can you specify hardware platform and software version here? I am not seeing what you're seeing, the config I sent blocks a BGP port scan in nmap, and prevents BGP peering to anything other than the specified IP. I am testing on Nexus 9500 with 10.1 NXOS; I suspect you are using some other platform eg n5k or n7k? I may be able to try it out here depending on what you're using. Thanks, Tim -Original Message- From: Drew Weaver Sent: Thursday, June 3, 2021 12:37 PM To: Tim Stevenson (tstevens) ; 'cisco-nsp@puck.nether.net' Subject: RE: Nexus Architecture question Hi Tim, I replied off-list with additional details but I wanted to reply on list to answer your specific questions: I copied the strict policy and then edited it with these classes at the top: policy-map type control-plane Test6 class custom-copp-system-p-bgp-allow police cir 19000 pps bc 128 packets conform transmit violate drop class custom-copp-system-p-bgp-deny police cir 0 pps bc 1 packets conform transmit violate drop These are the configured class-maps: Class-map type control-plane match-any custom-copp-system-p-bgp-allow match access-group name custom-copp-system-p-acl-bgp-allow Class-map type control-plane match-any custom-copp-system-p-bgp-deny match access-group name custom-copp-system-p-acl-bgp-deny These are the configured ACLs: IP access list custom-copp-system-p-acl-bgp-allow 3 permit tcp 192.168.1.2/32 gt 1023 any eq bgp 4 permit tcp 192.168.1.2/32 eq bgp any gt 1023 IP access list custom-copp-system-p-acl-bgp-deny 1 permit tcp any any eq bgp 10 permit tcp any gt 1023 any eq bgp 20 permit tcp any eq bgp any gt 1023 control-plane service-policy input Test6 The primary difference that I notice between the two configurations is that your allow is set to cos 7 and on mine it is not specified and yours is policing on bps and mine is policing on pps. (I am pretty certain that throughout the process I have tried it both ways). I find it a bit hard to follow that on this platform there is no way to write "just block everything that matches this class" you still have to give it a "burst" [which I obviously don't want] I understand that behind the scenes the control plane is supposed to "do the right thing" but it is just difficult to understand that from reading it. With this configuration applied: 1) Any host that NMAPs the device sees 179 open. 2) BGP sessions establish (if configured on the Nexus) for hosts that are not 192.168.1.2/32 Is there any way to find out what class traffic from a specific SRC IP or to a specific port is getting caught at in a CoPP policy? I suspect that the TCP/179 traffic is just bypassing these classes altogether and most likely are getting handled further down in the CoPP policy but I haven't been able to prove that. Thanks so much. -Drew -Original Message- From: Tim Stevenson (tstevens) Sent: Wednesday, June 2, 2021 5:31 PM To: Drew Weaver ; 'cisco-nsp@puck.nether.net' Subject: RE: Nexus Architecture question Hi Drew, In answer to your question about BGP, the BGP process runs only on the supervisor engine, it does not run on the linecards or anywhere else. It's a single process, not a per-interface process or anything like that. Curious how exactly you are configuring CoPP to filter this? With default CoPP, I get an "open/tcpwrapped" (green) response on TCP 179; I was able to get a "filtered" (red) response by adding a CoPP class that matches on BGP and polices to a CIR of 0. I preceeded that class with a class that matches on a specific neighborship - that BGP neighborship is successfully established while nmap still returns as filtered from my host. ip access-list allow-bgp 10 permit tcp 10.1.1.1/32 gt 1023 10.1.1.2/32 eq bgp 20 permit tcp 10.1.1.2/32 eq bgp 10.1.1.1/32 gt 1023 ip access-list drop-bgp 10 permit tcp any any eq bgp 20 permit tcp any eq bgp any ! class-map type control-plane match-any allow-bgp match access-group name allow-bgp class-map type control-plane match-any drop-bgp match access-group name drop-bgp ! policy-map type control-plane test-copp-policy-strict class allow-bgp set cos 7 police cir 36000 kbps bc 128 bytes conform transmit violate dr
Re: [c-nsp] Nexus Architecture question
Sure, N9K-SUP-B version 9.3(6) Also tried: 7.0(3)I7(8) Thanks, -Drew -Original Message- From: Tim Stevenson (tstevens) Sent: Friday, June 4, 2021 4:40 PM To: Drew Weaver ; 'cisco-nsp@puck.nether.net' Subject: RE: Nexus Architecture question Hi Drew, Can you specify hardware platform and software version here? I am not seeing what you're seeing, the config I sent blocks a BGP port scan in nmap, and prevents BGP peering to anything other than the specified IP. I am testing on Nexus 9500 with 10.1 NXOS; I suspect you are using some other platform eg n5k or n7k? I may be able to try it out here depending on what you're using. Thanks, Tim -Original Message- From: Drew Weaver Sent: Thursday, June 3, 2021 12:37 PM To: Tim Stevenson (tstevens) ; 'cisco-nsp@puck.nether.net' Subject: RE: Nexus Architecture question Hi Tim, I replied off-list with additional details but I wanted to reply on list to answer your specific questions: I copied the strict policy and then edited it with these classes at the top: policy-map type control-plane Test6 class custom-copp-system-p-bgp-allow police cir 19000 pps bc 128 packets conform transmit violate drop class custom-copp-system-p-bgp-deny police cir 0 pps bc 1 packets conform transmit violate drop These are the configured class-maps: Class-map type control-plane match-any custom-copp-system-p-bgp-allow match access-group name custom-copp-system-p-acl-bgp-allow Class-map type control-plane match-any custom-copp-system-p-bgp-deny match access-group name custom-copp-system-p-acl-bgp-deny These are the configured ACLs: IP access list custom-copp-system-p-acl-bgp-allow 3 permit tcp 192.168.1.2/32 gt 1023 any eq bgp 4 permit tcp 192.168.1.2/32 eq bgp any gt 1023 IP access list custom-copp-system-p-acl-bgp-deny 1 permit tcp any any eq bgp 10 permit tcp any gt 1023 any eq bgp 20 permit tcp any eq bgp any gt 1023 control-plane service-policy input Test6 The primary difference that I notice between the two configurations is that your allow is set to cos 7 and on mine it is not specified and yours is policing on bps and mine is policing on pps. (I am pretty certain that throughout the process I have tried it both ways). I find it a bit hard to follow that on this platform there is no way to write "just block everything that matches this class" you still have to give it a "burst" [which I obviously don't want] I understand that behind the scenes the control plane is supposed to "do the right thing" but it is just difficult to understand that from reading it. With this configuration applied: 1) Any host that NMAPs the device sees 179 open. 2) BGP sessions establish (if configured on the Nexus) for hosts that are not 192.168.1.2/32 Is there any way to find out what class traffic from a specific SRC IP or to a specific port is getting caught at in a CoPP policy? I suspect that the TCP/179 traffic is just bypassing these classes altogether and most likely are getting handled further down in the CoPP policy but I haven't been able to prove that. Thanks so much. -Drew -Original Message- From: Tim Stevenson (tstevens) Sent: Wednesday, June 2, 2021 5:31 PM To: Drew Weaver ; 'cisco-nsp@puck.nether.net' Subject: RE: Nexus Architecture question Hi Drew, In answer to your question about BGP, the BGP process runs only on the supervisor engine, it does not run on the linecards or anywhere else. It's a single process, not a per-interface process or anything like that. Curious how exactly you are configuring CoPP to filter this? With default CoPP, I get an "open/tcpwrapped" (green) response on TCP 179; I was able to get a "filtered" (red) response by adding a CoPP class that matches on BGP and polices to a CIR of 0. I preceeded that class with a class that matches on a specific neighborship - that BGP neighborship is successfully established while nmap still returns as filtered from my host. ip access-list allow-bgp 10 permit tcp 10.1.1.1/32 gt 1023 10.1.1.2/32 eq bgp 20 permit tcp 10.1.1.2/32 eq bgp 10.1.1.1/32 gt 1023 ip access-list drop-bgp 10 permit tcp any any eq bgp 20 permit tcp any eq bgp any ! class-map type control-plane match-any allow-bgp match access-group name allow-bgp class-map type control-plane match-any drop-bgp match access-group name drop-bgp ! policy-map type control-plane test-copp-policy-strict class allow-bgp set cos 7 police cir 36000 kbps bc 128 bytes conform transmit violate drop class drop-bgp police cir 0 bps bc 32000 bytes conform transmit violate drop Hope that helps, Tim -Original Message- From: cisco-nsp On Behalf Of Drew Weaver Sent: Wednesday, June 2, 2021 6:41 AM To: 'cisco-nsp@puck.nether.net' Subject: [c-nsp] Nexus Architecture qu
Re: [c-nsp] Nexus Architecture question
On Thu, 3 Jun 2021 at 22:46, Drew Weaver wrote: > IP access list custom-copp-system-p-acl-bgp-allow > 3 permit tcp 192.168.1.2/32 gt 1023 any eq bgp > 4 permit tcp 192.168.1.2/32 eq bgp any gt 1023 > > IP access list custom-copp-system-p-acl-bgp-deny > 1 permit tcp any any eq bgp > 10 permit tcp any gt 1023 any eq bgp > 20 permit tcp any eq bgp any gt 1023 a) there is no reason to limit far-end ephemeral range (added cost, complexity and it might break some broken implementation causing work on your end, while you don't actually care if your customer uses broken implementation). b) there is good reason to limit near-end ephemeral range to actual ephemeral range, as there can be local ports listening at >1024 XR appears to use an ephemeral range of 15000-57343, unfortunately as far as i can see it is not documented therefore not guaranteed across upgrades :( -- ++ytti ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/
Re: [c-nsp] Nexus Architecture question
--- Begin Message --- Hi Drew, Can you specify hardware platform and software version here? I am not seeing what you're seeing, the config I sent blocks a BGP port scan in nmap, and prevents BGP peering to anything other than the specified IP. I am testing on Nexus 9500 with 10.1 NXOS; I suspect you are using some other platform eg n5k or n7k? I may be able to try it out here depending on what you're using. Thanks, Tim -Original Message- From: Drew Weaver Sent: Thursday, June 3, 2021 12:37 PM To: Tim Stevenson (tstevens) ; 'cisco-nsp@puck.nether.net' Subject: RE: Nexus Architecture question Hi Tim, I replied off-list with additional details but I wanted to reply on list to answer your specific questions: I copied the strict policy and then edited it with these classes at the top: policy-map type control-plane Test6 class custom-copp-system-p-bgp-allow police cir 19000 pps bc 128 packets conform transmit violate drop class custom-copp-system-p-bgp-deny police cir 0 pps bc 1 packets conform transmit violate drop These are the configured class-maps: Class-map type control-plane match-any custom-copp-system-p-bgp-allow match access-group name custom-copp-system-p-acl-bgp-allow Class-map type control-plane match-any custom-copp-system-p-bgp-deny match access-group name custom-copp-system-p-acl-bgp-deny These are the configured ACLs: IP access list custom-copp-system-p-acl-bgp-allow 3 permit tcp 192.168.1.2/32 gt 1023 any eq bgp 4 permit tcp 192.168.1.2/32 eq bgp any gt 1023 IP access list custom-copp-system-p-acl-bgp-deny 1 permit tcp any any eq bgp 10 permit tcp any gt 1023 any eq bgp 20 permit tcp any eq bgp any gt 1023 control-plane service-policy input Test6 The primary difference that I notice between the two configurations is that your allow is set to cos 7 and on mine it is not specified and yours is policing on bps and mine is policing on pps. (I am pretty certain that throughout the process I have tried it both ways). I find it a bit hard to follow that on this platform there is no way to write "just block everything that matches this class" you still have to give it a "burst" [which I obviously don't want] I understand that behind the scenes the control plane is supposed to "do the right thing" but it is just difficult to understand that from reading it. With this configuration applied: 1) Any host that NMAPs the device sees 179 open. 2) BGP sessions establish (if configured on the Nexus) for hosts that are not 192.168.1.2/32 Is there any way to find out what class traffic from a specific SRC IP or to a specific port is getting caught at in a CoPP policy? I suspect that the TCP/179 traffic is just bypassing these classes altogether and most likely are getting handled further down in the CoPP policy but I haven't been able to prove that. Thanks so much. -Drew -Original Message- From: Tim Stevenson (tstevens) Sent: Wednesday, June 2, 2021 5:31 PM To: Drew Weaver ; 'cisco-nsp@puck.nether.net' Subject: RE: Nexus Architecture question Hi Drew, In answer to your question about BGP, the BGP process runs only on the supervisor engine, it does not run on the linecards or anywhere else. It's a single process, not a per-interface process or anything like that. Curious how exactly you are configuring CoPP to filter this? With default CoPP, I get an "open/tcpwrapped" (green) response on TCP 179; I was able to get a "filtered" (red) response by adding a CoPP class that matches on BGP and polices to a CIR of 0. I preceeded that class with a class that matches on a specific neighborship - that BGP neighborship is successfully established while nmap still returns as filtered from my host. ip access-list allow-bgp 10 permit tcp 10.1.1.1/32 gt 1023 10.1.1.2/32 eq bgp 20 permit tcp 10.1.1.2/32 eq bgp 10.1.1.1/32 gt 1023 ip access-list drop-bgp 10 permit tcp any any eq bgp 20 permit tcp any eq bgp any ! class-map type control-plane match-any allow-bgp match access-group name allow-bgp class-map type control-plane match-any drop-bgp match access-group name drop-bgp ! policy-map type control-plane test-copp-policy-strict class allow-bgp set cos 7 police cir 36000 kbps bc 128 bytes conform transmit violate drop class drop-bgp police cir 0 bps bc 32000 bytes conform transmit violate drop Hope that helps, Tim -Original Message- From: cisco-nsp On Behalf Of Drew Weaver Sent: Wednesday, June 2, 2021 6:41 AM To: 'cisco-nsp@puck.nether.net' Subject: [c-nsp] Nexus Architecture question Has anyone seen a document from Cisco that shows where various processes running on various Nexus switches actually run from? For example on a 9508 the nxapi runs in a Linux VM and in order to secure it you have to drop into the VM and use i
Re: [c-nsp] Nexus Architecture question
Hi Tim, I replied off-list with additional details but I wanted to reply on list to answer your specific questions: I copied the strict policy and then edited it with these classes at the top: policy-map type control-plane Test6 class custom-copp-system-p-bgp-allow police cir 19000 pps bc 128 packets conform transmit violate drop class custom-copp-system-p-bgp-deny police cir 0 pps bc 1 packets conform transmit violate drop These are the configured class-maps: Class-map type control-plane match-any custom-copp-system-p-bgp-allow match access-group name custom-copp-system-p-acl-bgp-allow Class-map type control-plane match-any custom-copp-system-p-bgp-deny match access-group name custom-copp-system-p-acl-bgp-deny These are the configured ACLs: IP access list custom-copp-system-p-acl-bgp-allow 3 permit tcp 192.168.1.2/32 gt 1023 any eq bgp 4 permit tcp 192.168.1.2/32 eq bgp any gt 1023 IP access list custom-copp-system-p-acl-bgp-deny 1 permit tcp any any eq bgp 10 permit tcp any gt 1023 any eq bgp 20 permit tcp any eq bgp any gt 1023 control-plane service-policy input Test6 The primary difference that I notice between the two configurations is that your allow is set to cos 7 and on mine it is not specified and yours is policing on bps and mine is policing on pps. (I am pretty certain that throughout the process I have tried it both ways). I find it a bit hard to follow that on this platform there is no way to write "just block everything that matches this class" you still have to give it a "burst" [which I obviously don't want] I understand that behind the scenes the control plane is supposed to "do the right thing" but it is just difficult to understand that from reading it. With this configuration applied: 1) Any host that NMAPs the device sees 179 open. 2) BGP sessions establish (if configured on the Nexus) for hosts that are not 192.168.1.2/32 Is there any way to find out what class traffic from a specific SRC IP or to a specific port is getting caught at in a CoPP policy? I suspect that the TCP/179 traffic is just bypassing these classes altogether and most likely are getting handled further down in the CoPP policy but I haven't been able to prove that. Thanks so much. -Drew -Original Message- From: Tim Stevenson (tstevens) Sent: Wednesday, June 2, 2021 5:31 PM To: Drew Weaver ; 'cisco-nsp@puck.nether.net' Subject: RE: Nexus Architecture question Hi Drew, In answer to your question about BGP, the BGP process runs only on the supervisor engine, it does not run on the linecards or anywhere else. It's a single process, not a per-interface process or anything like that. Curious how exactly you are configuring CoPP to filter this? With default CoPP, I get an "open/tcpwrapped" (green) response on TCP 179; I was able to get a "filtered" (red) response by adding a CoPP class that matches on BGP and polices to a CIR of 0. I preceeded that class with a class that matches on a specific neighborship - that BGP neighborship is successfully established while nmap still returns as filtered from my host. ip access-list allow-bgp 10 permit tcp 10.1.1.1/32 gt 1023 10.1.1.2/32 eq bgp 20 permit tcp 10.1.1.2/32 eq bgp 10.1.1.1/32 gt 1023 ip access-list drop-bgp 10 permit tcp any any eq bgp 20 permit tcp any eq bgp any ! class-map type control-plane match-any allow-bgp match access-group name allow-bgp class-map type control-plane match-any drop-bgp match access-group name drop-bgp ! policy-map type control-plane test-copp-policy-strict class allow-bgp set cos 7 police cir 36000 kbps bc 128 bytes conform transmit violate drop class drop-bgp police cir 0 bps bc 32000 bytes conform transmit violate drop Hope that helps, Tim -Original Message- From: cisco-nsp On Behalf Of Drew Weaver Sent: Wednesday, June 2, 2021 6:41 AM To: 'cisco-nsp@puck.nether.net' Subject: [c-nsp] Nexus Architecture question Has anyone seen a document from Cisco that shows where various processes running on various Nexus switches actually run from? For example on a 9508 the nxapi runs in a Linux VM and in order to secure it you have to drop into the VM and use iptables. I am trying to figure out where the BGP process lives (for lack of a better word). Does it run on the line cards? In the control plane? Both? Does it vary depending on which model and which line cards? The reason I am asking is because I've noticed that no matter what I do I cannot seem to "close" the BGP port by using CoPP. It always shows up as being open when doing a port scan against the system using NMAP. I know that the switch should not establish a connection with random hosts but I really am getting hung up on it being 'scannable'/visible at all. ___
Re: [c-nsp] Nexus Architecture question
--- Begin Message --- Hi Drew, In answer to your question about BGP, the BGP process runs only on the supervisor engine, it does not run on the linecards or anywhere else. It's a single process, not a per-interface process or anything like that. Curious how exactly you are configuring CoPP to filter this? With default CoPP, I get an "open/tcpwrapped" (green) response on TCP 179; I was able to get a "filtered" (red) response by adding a CoPP class that matches on BGP and polices to a CIR of 0. I preceeded that class with a class that matches on a specific neighborship - that BGP neighborship is successfully established while nmap still returns as filtered from my host. ip access-list allow-bgp 10 permit tcp 10.1.1.1/32 gt 1023 10.1.1.2/32 eq bgp 20 permit tcp 10.1.1.2/32 eq bgp 10.1.1.1/32 gt 1023 ip access-list drop-bgp 10 permit tcp any any eq bgp 20 permit tcp any eq bgp any ! class-map type control-plane match-any allow-bgp match access-group name allow-bgp class-map type control-plane match-any drop-bgp match access-group name drop-bgp ! policy-map type control-plane test-copp-policy-strict class allow-bgp set cos 7 police cir 36000 kbps bc 128 bytes conform transmit violate drop class drop-bgp police cir 0 bps bc 32000 bytes conform transmit violate drop Hope that helps, Tim -Original Message- From: cisco-nsp On Behalf Of Drew Weaver Sent: Wednesday, June 2, 2021 6:41 AM To: 'cisco-nsp@puck.nether.net' Subject: [c-nsp] Nexus Architecture question Has anyone seen a document from Cisco that shows where various processes running on various Nexus switches actually run from? For example on a 9508 the nxapi runs in a Linux VM and in order to secure it you have to drop into the VM and use iptables. I am trying to figure out where the BGP process lives (for lack of a better word). Does it run on the line cards? In the control plane? Both? Does it vary depending on which model and which line cards? The reason I am asking is because I've noticed that no matter what I do I cannot seem to "close" the BGP port by using CoPP. It always shows up as being open when doing a port scan against the system using NMAP. I know that the switch should not establish a connection with random hosts but I really am getting hung up on it being 'scannable'/visible at all. ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/ --- End Message --- ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/
Re: [c-nsp] Nexus Architecture question
Absolutely, iACLs can also prevent access to NXAPI and SNMP but those aren't the most central place to put rules like that. If iACLs are the only way to prevent access to BGP fine, but what would be the technical explanation for that? Does BGP run a separate instance on each L3 interface or is there a central process (somewhere)? Thanks, -Drew -Original Message- From: Dobbins, Roland Sent: Wednesday, June 2, 2021 2:30 PM To: Drew Weaver Cc: cisco-nsp@puck.nether.net Subject: Re: [c-nsp] Nexus Architecture question > On Jun 2, 2021, at 20:46, Drew Weaver wrote: > > The reason I am asking is because I've noticed that no matter what I do I > cannot seem to "close" the BGP port by using CoPP. iACLs can accomplish the goal, yes? --- roland.dobb...@netscout.com ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/
Re: [c-nsp] Nexus Architecture question
> On Jun 2, 2021, at 20:46, Drew Weaver wrote: > > The reason I am asking is because I've noticed that no matter what I do I > cannot seem to "close" the BGP port by using CoPP. iACLs can accomplish the goal, yes? --- roland.dobb...@netscout.com ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/
[c-nsp] Nexus Architecture question
Has anyone seen a document from Cisco that shows where various processes running on various Nexus switches actually run from? For example on a 9508 the nxapi runs in a Linux VM and in order to secure it you have to drop into the VM and use iptables. I am trying to figure out where the BGP process lives (for lack of a better word). Does it run on the line cards? In the control plane? Both? Does it vary depending on which model and which line cards? The reason I am asking is because I've noticed that no matter what I do I cannot seem to "close" the BGP port by using CoPP. It always shows up as being open when doing a port scan against the system using NMAP. I know that the switch should not establish a connection with random hosts but I really am getting hung up on it being 'scannable'/visible at all. ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/
