Re: [j-nsp] Using a QFX5100 without QFabric?
MPLS is now supported on IRB on QFX5100: https://www.juniper.net/documentation/en_US/junos/topics/reference/general/mpls-limitations-qfx-series.html#jd0e57 On Fri, Oct 27, 2017 at 3:50 PM, Andrey Kostinwrote: > Chris Wopat писал 25.10.2017 13:00: > >> On 10/24/2017 05:30 PM, Vincent Bernat wrote: >> >>> ❦ 24 octobre 2017 14:29 -0400, Andrey Kostin : >>> >>> > Straight up saying "don't put public IPs on them" doesn't seem like >> the best advice to me. You can certainly do this, we do and it's fine. >> When you craft your RE protection filter you just have to squeeze a >> bit more space here or there compared to say, an MX filter. You should >> have this enabled weather you're using public IPs or not. >> >> Regarding TCAM programming, it's loud and clear when this happens via >> a console message and a sev0 syslog message. >> > > Yes, that's true, and we spend a decent amount of time packing lo0 filters > in a tiny TCAM after discovered that filter input-list silently allows > everything except the first filter and doesn't generate any complaint. > So, no objection for public IPs but only careful filter planning required. > > -- > Kind regards, > Andrey > > ___ > juniper-nsp mailing list juniper-nsp@puck.nether.net > https://puck.nether.net/mailman/listinfo/juniper-nsp > ___ juniper-nsp mailing list juniper-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/juniper-nsp
Re: [j-nsp] Using a QFX5100 without QFabric?
Chris Wopat писал 25.10.2017 13:00: On 10/24/2017 05:30 PM, Vincent Bernat wrote: ❦ 24 octobre 2017 14:29 -0400, Andrey Kostin: Straight up saying "don't put public IPs on them" doesn't seem like the best advice to me. You can certainly do this, we do and it's fine. When you craft your RE protection filter you just have to squeeze a bit more space here or there compared to say, an MX filter. You should have this enabled weather you're using public IPs or not. Regarding TCAM programming, it's loud and clear when this happens via a console message and a sev0 syslog message. Yes, that's true, and we spend a decent amount of time packing lo0 filters in a tiny TCAM after discovered that filter input-list silently allows everything except the first filter and doesn't generate any complaint. So, no objection for public IPs but only careful filter planning required. -- Kind regards, Andrey ___ juniper-nsp mailing list juniper-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/juniper-nsp
Re: [j-nsp] Using a QFX5100 without QFabric?
Vincent Bernat писал 24.10.2017 18:30: ❦ 24 octobre 2017 14:29 -0400, Andrey Kostin: QFX5100 are good as L2 devices for aggregation, we use them in virtual-chassis. But be careful with planning any L3 services on them. First, don't put public IPs on them because TCAM for filters is tiny and programmed in a tricky for understanding way. As a result everything that doesn't fit in TCAM is silently allowed. We observed that lo0 filters were "bypassed" this way and switch was exposed to continuous brute-force attack. That's scary! I remember having a commit error when I set too many filters (in fact, too many source/destination combination, solved by removing either source or destination from the filter), so there are some checks in place. Which version were you using when you got the problem? Is there an easy way to check if we are hit by that? At that moment (Feb 2016) it was 13.2X51-D35.3. Is I can see from the link posted in the thread, MPLS on IRB is not supported yet, probably hardware limitation. Here is the conclusion from JTAC case: Problem Description : We use common set of filters on all our juniper devices to protect control plane and it turnes out there is a strange problem with filter on QFX switches. When that input filter list is applied then at least ports tcp/22 and tcp/179 are world-wide open. Issue: Filter was not getting programmed in TCAM: Action taken: As per our latest communication, we have identified two reasons behind the filters not getting programmed First, the filter entries exceeded the maximum TCAM entries. Second, we observed the the QFX platforms do not support input-list. Although the config gets committed without any error, only the first filter gets programmed in TCAM. We also provided a sample configuration to demonstrate the ssh filter. JTAC engineer's examples provided: I have tried the following configs in the lab under 13.2X51-D35 and 14.1X53-D30 and have observed the following: Config independent of the group: set interfaces lo0 unit 0 family inet filter input-list [ accept-ftp accept-ssh ] Config within group: set groups common:lo-filter interfaces lo0 unit 0 family inet filter input-list accept-ftp set groups common:lo-filter interfaces lo0 unit 0 family inet filter input-list accept-ssh In both cases, the configuration goes through without any error but only the first filter (accept-ftp) actually gets programmed in the PFE programs as can observed below: TFXPC0(vty)# show filter Program Filters: --- Index Dir CntText Bss Name -- -- -- -- Term Filters: IndexSemantic Name -- -- 1 Classicaccept-ftp 2 Classicaccept-ssh 3 Classiclo0.0-i 17000 Classic__default_arp_policer__ 16777216 Classicfnp-filter-level-all TFXPC0(vty)# show filter hw 3 show_term_info == Filter index : 3 == - Filter name : lo0.0-i + Programmed: YES + BD ID : 184 + Total TCAM entries available: 1528 + Total TCAM entries needed : 8 + Term Expansion: - Term1: will expand to 1 term : Name "accept-ftp-0" - Term2: will expand to 1 term : Name "accept-ftp-1" + Term TCAM entry requirements: - Term1: needs 4 TCAM entries: Name "accept-ftp-0" - Term2: needs 4 TCAM entries: Name "accept-ftp-1" + Total TCAM entries available: 1528 + Total TCAM entries needed : 8 Even the counters only show the counters for the first filter (accept-filter) and not those for the following filters (accept-ssh) in the input-list. The following is missing count-accept-ssh-lo0.0-i . Second thing I can recall is that MPLS works only on physical interfaces, not irb. And finally I had very mixed results when tried to PIM multicast routing between irb interfaces and have to give up and pass L2 to a router, didn't try it on physical ports though. I had also some bad experience with IRB on QFX5100. For example, unnumbered interfaces don't work on IRB. Also, I have also already related here my troubles with IRB, routing daemons and MC-LAG. For some reasons, it seems many features don't play well with IRB (at least on 14.1X53 train). I am now using them as L2 switches and as BGP RR (but no routing) and so far, no problems. -- Kind regards, Andrey ___ juniper-nsp mailing list juniper-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/juniper-nsp
Re: [j-nsp] Using a QFX5100 without QFabric?
On 10/24/2017 05:30 PM, Vincent Bernat wrote: ❦ 24 octobre 2017 14:29 -0400, Andrey Kostin: QFX5100 are good as L2 devices for aggregation, we use them in virtual-chassis. But be careful with planning any L3 services on them. First, don't put public IPs on them because TCAM for filters is tiny and programmed in a tricky for understanding way. As a result everything that doesn't fit in TCAM is silently allowed. We observed that lo0 filters were "bypassed" this way and switch was exposed to continuous brute-force attack. That's scary! I remember having a commit error when I set too many filters (in fact, too many source/destination combination, solved by removing either source or destination from the filter), so there are some checks in place. Which version were you using when you got the problem? Is there an easy way to check if we are hit by that? Straight up saying "don't put public IPs on them" doesn't seem like the best advice to me. You can certainly do this, we do and it's fine. When you craft your RE protection filter you just have to squeeze a bit more space here or there compared to say, an MX filter. You should have this enabled weather you're using public IPs or not. Regarding TCAM programming, it's loud and clear when this happens via a console message and a sev0 syslog message. You can check current TCAM levels with `show pfe filter hw summary`. If you need to know details you can find them via fpc shell: > start shell % vty fpc0 TFXPC0(vty)# show filter Program Filters: --- Index Dir CntText Bss Name -- -- -- -- Term Filters: IndexSemanticName 1 Classic accept-only 2 Classic classify-accept 3 Classic protect-re TFXPC0(vty)# show filter hw 3 show_term_info == Filter index : 3 == --Chris ___ juniper-nsp mailing list juniper-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/juniper-nsp
Re: [j-nsp] Using a QFX5100 without QFabric?
On Wed, Oct 25, 2017 at 8:49 AM, Aaron Gouldwrote: > I let someone else worry about dollars… but sounds like you have a point > > You can do mpls l2circuit (martini) in a qfx ? I was under the impression > the qfx did no mpls at all. > > https://www.juniper.net/documentation/en_US/junos/topics/concept/mpls-features-qfx-series-overview.html ___ juniper-nsp mailing list juniper-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/juniper-nsp
Re: [j-nsp] Using a QFX5100 without QFabric?
I let someone else worry about dollars… but sounds like you have a point You can do mpls l2circuit (martini) in a qfx ? I was under the impression the qfx did no mpls at all. -Aaron From: Joe Freeman [mailto:j...@netbyjoe.com] Sent: Tuesday, October 24, 2017 1:21 PM To: Aaron Gould Cc: mlfre...@mtu.edu; Karl Gerhard; Juniper List Subject: Re: [j-nsp] Using a QFX5100 without QFabric? How do you handle 10G port licensing on the 5048? That gets expensive quickly. I've got about 75 qfx's deployed as PE devices right now because of the 5048 port licenses. The major limitation of the qfx as a PE device is that it doesn't support VPLS. It does however do EVPN over vxlan, which can be stitched to a vpls instance if needed on an MX, at least according to Juniper. I've not yet tried it. On the very few instances where I've absolutely had to deliver a VPLS type service, I've been able to bring L2circuits back to a 480 and stitch them all to a bridge-domain there. Not optimal, but it works. Joe The qfx's do L3vpn and l2circuits nicely, with RSVP/LDP, BGP, and ISIS. On Tue, Oct 24, 2017 at 9:44 AM, Aaron Gould <aar...@gvtc.com> wrote: Not to change subject too much, but, In case you are wanting to extend your mpls cloud (I'm assuming your MX core is mpls-enabled) further out into the aggregation/access edge, you could go with the qfx-5100 cousin... acx5048. I've been pretty pleased with them. I've deployed 30 or 40 of these now in my network with as cisco asr9k core. -Aaron ___ juniper-nsp mailing list juniper-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/juniper-nsp ___ juniper-nsp mailing list juniper-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/juniper-nsp
Re: [j-nsp] Using a QFX5100 without QFabric?
❦ 24 octobre 2017 14:29 -0400, Andrey Kostin: > QFX5100 are good as L2 devices for aggregation, we use them in > virtual-chassis. But be careful with planning any L3 services on > them. First, don't put public IPs on them because TCAM for filters is > tiny and programmed in a tricky for understanding way. As a result > everything that doesn't fit in TCAM is silently allowed. We observed > that lo0 filters were "bypassed" this way and switch was exposed to > continuous brute-force attack. That's scary! I remember having a commit error when I set too many filters (in fact, too many source/destination combination, solved by removing either source or destination from the filter), so there are some checks in place. Which version were you using when you got the problem? Is there an easy way to check if we are hit by that? > Second thing I can recall is that MPLS works only on physical > interfaces, not irb. And finally I had very mixed results when tried > to PIM multicast routing between irb interfaces and have to give up > and pass L2 to a router, didn't try it on physical ports though. I had also some bad experience with IRB on QFX5100. For example, unnumbered interfaces don't work on IRB. Also, I have also already related here my troubles with IRB, routing daemons and MC-LAG. For some reasons, it seems many features don't play well with IRB (at least on 14.1X53 train). I am now using them as L2 switches and as BGP RR (but no routing) and so far, no problems. -- Don't go around saying the world owes you a living. The world owes you nothing. It was here first. -- Mark Twain ___ juniper-nsp mailing list juniper-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/juniper-nsp
Re: [j-nsp] Using a QFX5100 without QFabric?
Without ASCII art: We have P(MX), a P(vMX), a PE1 (QFX5100), and PE2 (QFX5100), all with ISIS, MPLS, RSVP/LDP, BGP underlay, cluster and multipath. The (EVPN) broadcast is handled by the P's but once that discovery is done, the traffic passes between the PE's without bouncing thru the P's. Good enough for what it is. Must be the same deal with VPLS. Caveats: Can't mix EVPN and/or L2 and/or CCC on the same port. Right now EVPN+CCC works for our lab, but it must be a fluke. - Alain Hebertaheb...@pubnix.net PubNIX Inc. 50 boul. St-Charles P.O. Box 26770 Beaconsfield, Quebec H9W 6G7 Tel: 514-990-5911 http://www.pubnix.netFax: 514-990-9443 On 10/24/17 14:20, Joe Freeman wrote: How do you handle 10G port licensing on the 5048? That gets expensive quickly. I've got about 75 qfx's deployed as PE devices right now because of the 5048 port licenses. The major limitation of the qfx as a PE device is that it doesn't support VPLS. It does however do EVPN over vxlan, which can be stitched to a vpls instance if needed on an MX, at least according to Juniper. I've not yet tried it. On the very few instances where I've absolutely had to deliver a VPLS type service, I've been able to bring L2circuits back to a 480 and stitch them all to a bridge-domain there. Not optimal, but it works. Joe The qfx's do L3vpn and l2circuits nicely, with RSVP/LDP, BGP, and ISIS. On Tue, Oct 24, 2017 at 9:44 AM, Aaron Gouldwrote: Not to change subject too much, but, In case you are wanting to extend your mpls cloud (I'm assuming your MX core is mpls-enabled) further out into the aggregation/access edge, you could go with the qfx-5100 cousin... acx5048. I've been pretty pleased with them. I've deployed 30 or 40 of these now in my network with as cisco asr9k core. -Aaron ___ juniper-nsp mailing list juniper-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/juniper-nsp ___ juniper-nsp mailing list juniper-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/juniper-nsp ___ juniper-nsp mailing list juniper-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/juniper-nsp
Re: [j-nsp] Using a QFX5100 without QFabric?
Hi, We have a stub vrf with Transit on them, the solution is a very good set of filters on lo0 input. - Alain Hebertaheb...@pubnix.net PubNIX Inc. 50 boul. St-Charles P.O. Box 26770 Beaconsfield, Quebec H9W 6G7 Tel: 514-990-5911 http://www.pubnix.netFax: 514-990-9443 On 10/24/17 14:29, Andrey Kostin wrote: QFX5100 are good as L2 devices for aggregation, we use them in virtual-chassis. But be careful with planning any L3 services on them. First, don't put public IPs on them because TCAM for filters is tiny and programmed in a tricky for understanding way. As a result everything that doesn't fit in TCAM is silently allowed. We observed that lo0 filters were "bypassed" this way and switch was exposed to continuous brute-force attack. Second thing I can recall is that MPLS works only on physical interfaces, not irb. And finally I had very mixed results when tried to PIM multicast routing between irb interfaces and have to give up and pass L2 to a router, didn't try it on physical ports though. Kind regards, Andrey Kostin Matt Freitag писал 24.10.2017 09:26: Karl, we're also looking at QFX5100-48S switches for our aggregation. I actually have one in place doing aggregation and routing and the only "big" change I found is the DHCP forwarder config is not remotely similar to the forwarding-options helpers bootp config we've been using to forward DHCP on our MX480 core. But that only counts if you do routing and DHCP forwarding at the QFX. But, if you want to do routing and DHCP forwarding on this, any forwarding in the default routing instance goes under forwarding-options dhcp-relay and any DHCP forwarding in a non-default routing instance goes under routing-instances INSTANCE-NAME forwarding-options dhcp-relay. There are a ton of DHCP relay options but we found we just need a server group that contains all our DHCP servers and an interface group that ties an interface to a server group. Again I only bring the DHCP relay stuff up because we've been using forwarding-options helpers bootp on our MX's to do DHCP forwarding and the QFX explicitly disallows that in favor of the dhcp-relay. Other than that initial confusion we've not had a problem and I'm very interested in any issues you hear of. This QFX I'm talking about runs Junos 14.1X53-D40.8. I'm also very interested in any other issues people have had doing this. Matt Freitag Network Engineer Information Technology Michigan Technological University (906) 487-3696 <%28906%29%20487-3696> https://www.mtu.edu/ https://www.mtu.edu/it On Tue, Oct 24, 2017 at 8:41 AM, Karl Gerhardwrote: Hello we're thinking about buying a few QFX5100 as they are incredibly cheap on the refurbished market - sometimes even cheaper than a much older EX4550. Are there any caveats when using the QFX5100-48S as a normal aggregation switch without QFabric? We have a pretty basic setup of Access (EX), Aggregation (EX or QFX) and Core (MX). We're only switching at our aggregation layer but we would like to have options for the future. Regards Karl ___ juniper-nsp mailing list juniper-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/juniper-nsp ___ juniper-nsp mailing list juniper-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/juniper-nsp ___ juniper-nsp mailing list juniper-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/juniper-nsp ___ juniper-nsp mailing list juniper-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/juniper-nsp
Re: [j-nsp] Using a QFX5100 without QFabric?
QFX5100 are good as L2 devices for aggregation, we use them in virtual-chassis. But be careful with planning any L3 services on them. First, don't put public IPs on them because TCAM for filters is tiny and programmed in a tricky for understanding way. As a result everything that doesn't fit in TCAM is silently allowed. We observed that lo0 filters were "bypassed" this way and switch was exposed to continuous brute-force attack. Second thing I can recall is that MPLS works only on physical interfaces, not irb. And finally I had very mixed results when tried to PIM multicast routing between irb interfaces and have to give up and pass L2 to a router, didn't try it on physical ports though. Kind regards, Andrey Kostin Matt Freitag писал 24.10.2017 09:26: Karl, we're also looking at QFX5100-48S switches for our aggregation. I actually have one in place doing aggregation and routing and the only "big" change I found is the DHCP forwarder config is not remotely similar to the forwarding-options helpers bootp config we've been using to forward DHCP on our MX480 core. But that only counts if you do routing and DHCP forwarding at the QFX. But, if you want to do routing and DHCP forwarding on this, any forwarding in the default routing instance goes under forwarding-options dhcp-relay and any DHCP forwarding in a non-default routing instance goes under routing-instances INSTANCE-NAME forwarding-options dhcp-relay. There are a ton of DHCP relay options but we found we just need a server group that contains all our DHCP servers and an interface group that ties an interface to a server group. Again I only bring the DHCP relay stuff up because we've been using forwarding-options helpers bootp on our MX's to do DHCP forwarding and the QFX explicitly disallows that in favor of the dhcp-relay. Other than that initial confusion we've not had a problem and I'm very interested in any issues you hear of. This QFX I'm talking about runs Junos 14.1X53-D40.8. I'm also very interested in any other issues people have had doing this. Matt Freitag Network Engineer Information Technology Michigan Technological University (906) 487-3696 <%28906%29%20487-3696> https://www.mtu.edu/ https://www.mtu.edu/it On Tue, Oct 24, 2017 at 8:41 AM, Karl Gerhardwrote: Hello we're thinking about buying a few QFX5100 as they are incredibly cheap on the refurbished market - sometimes even cheaper than a much older EX4550. Are there any caveats when using the QFX5100-48S as a normal aggregation switch without QFabric? We have a pretty basic setup of Access (EX), Aggregation (EX or QFX) and Core (MX). We're only switching at our aggregation layer but we would like to have options for the future. Regards Karl ___ juniper-nsp mailing list juniper-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/juniper-nsp ___ juniper-nsp mailing list juniper-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/juniper-nsp ___ juniper-nsp mailing list juniper-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/juniper-nsp
Re: [j-nsp] Using a QFX5100 without QFabric?
How do you handle 10G port licensing on the 5048? That gets expensive quickly. I've got about 75 qfx's deployed as PE devices right now because of the 5048 port licenses. The major limitation of the qfx as a PE device is that it doesn't support VPLS. It does however do EVPN over vxlan, which can be stitched to a vpls instance if needed on an MX, at least according to Juniper. I've not yet tried it. On the very few instances where I've absolutely had to deliver a VPLS type service, I've been able to bring L2circuits back to a 480 and stitch them all to a bridge-domain there. Not optimal, but it works. Joe The qfx's do L3vpn and l2circuits nicely, with RSVP/LDP, BGP, and ISIS. On Tue, Oct 24, 2017 at 9:44 AM, Aaron Gouldwrote: > Not to change subject too much, but, In case you are wanting to extend your > mpls cloud (I'm assuming your MX core is mpls-enabled) further out into the > aggregation/access edge, you could go with the qfx-5100 cousin... acx5048. > I've been pretty pleased with them. > > I've deployed 30 or 40 of these now in my network with as cisco asr9k core. > > -Aaron > > > ___ > juniper-nsp mailing list juniper-nsp@puck.nether.net > https://puck.nether.net/mailman/listinfo/juniper-nsp > ___ juniper-nsp mailing list juniper-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/juniper-nsp
Re: [j-nsp] Using a QFX5100 without QFabric?
Not to change subject too much, but, In case you are wanting to extend your mpls cloud (I'm assuming your MX core is mpls-enabled) further out into the aggregation/access edge, you could go with the qfx-5100 cousin... acx5048. I've been pretty pleased with them. I've deployed 30 or 40 of these now in my network with as cisco asr9k core. -Aaron ___ juniper-nsp mailing list juniper-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/juniper-nsp
Re: [j-nsp] Using a QFX5100 without QFabric?
Karl, we're also looking at QFX5100-48S switches for our aggregation. I actually have one in place doing aggregation and routing and the only "big" change I found is the DHCP forwarder config is not remotely similar to the forwarding-options helpers bootp config we've been using to forward DHCP on our MX480 core. But that only counts if you do routing and DHCP forwarding at the QFX. But, if you want to do routing and DHCP forwarding on this, any forwarding in the default routing instance goes under forwarding-options dhcp-relay and any DHCP forwarding in a non-default routing instance goes under routing-instances INSTANCE-NAME forwarding-options dhcp-relay. There are a ton of DHCP relay options but we found we just need a server group that contains all our DHCP servers and an interface group that ties an interface to a server group. Again I only bring the DHCP relay stuff up because we've been using forwarding-options helpers bootp on our MX's to do DHCP forwarding and the QFX explicitly disallows that in favor of the dhcp-relay. Other than that initial confusion we've not had a problem and I'm very interested in any issues you hear of. This QFX I'm talking about runs Junos 14.1X53-D40.8. I'm also very interested in any other issues people have had doing this. Matt Freitag Network Engineer Information Technology Michigan Technological University (906) 487-3696 <%28906%29%20487-3696> https://www.mtu.edu/ https://www.mtu.edu/it On Tue, Oct 24, 2017 at 8:41 AM, Karl Gerhardwrote: > Hello > > we're thinking about buying a few QFX5100 as they are incredibly cheap on > the refurbished market - sometimes even cheaper than a much older EX4550. > > Are there any caveats when using the QFX5100-48S as a normal aggregation > switch without QFabric? We have a pretty basic setup of Access (EX), > Aggregation (EX or QFX) and Core (MX). We're only switching at our > aggregation layer but we would like to have options for the future. > > Regards > Karl > > ___ > juniper-nsp mailing list juniper-nsp@puck.nether.net > https://puck.nether.net/mailman/listinfo/juniper-nsp > ___ juniper-nsp mailing list juniper-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/juniper-nsp
[j-nsp] Using a QFX5100 without QFabric?
Hello we're thinking about buying a few QFX5100 as they are incredibly cheap on the refurbished market - sometimes even cheaper than a much older EX4550. Are there any caveats when using the QFX5100-48S as a normal aggregation switch without QFabric? We have a pretty basic setup of Access (EX), Aggregation (EX or QFX) and Core (MX). We're only switching at our aggregation layer but we would like to have options for the future. Regards Karl ___ juniper-nsp mailing list juniper-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/juniper-nsp