Re: [j-nsp] New 16port 10G Card and new MPC with 4x10G MIC Cards - coexistance of old DPCs and new Cards in same chassis -- looking for experience feedback
Hi, This is a pretty old thread, but the input might be useful. In the last couple of months we've been trying to mix DPCE-R-4XGEXFP with MPC-3D-16XGE-SFPP-R-B inside a MX960 chassis. The reason for using MPCs was scalability. The final result agreed upon together with Juniper was using the MX series platform either with DPCs or MPCs exclusively when expecting wire rate speed on all FPCs, 40Gbps/slot for the DPCs and 120Gbps/slot for the MPCs. Even mixing the different linecards with small traffic figures proved to be dangerous and ruined one of our maintenance windows. We are curious to find out what is the actual performance limit on the fixed MPC. Our highest record shows almost 60Gbps of traffic one way and 55Gbps the other way on a single MPC. Regards, Manu On Mon, Aug 30, 2010 at 1:56 AM, Richard A Steenbergen r...@e-gerbil.net wrote: On Sun, Aug 29, 2010 at 12:00:01PM -0700, Derick Winkworth wrote: so the possibility does exist that with a combination of newer fabric and newer line card (a line card with better MQ memory bandwidth), that MX might be able to push more traffic per slot... Sure, the chassis backplane is electrically capable of quite a bit, so if you keep upgrading the fabric and the cards you should be able to keep increasing bandwidth without forklifting the chassis for a long time. BTW, one more point of clarification on the MQ bandwidth limit. The 70G limit is actually the for bandwidth crossing the PFE in any direction, so the previously mentioned 35G fabric 35G wan example is actually based on the assumption of bidirectional traffic. To calculate the MQ usage you only want to count the packet ONCE per PFE crossing (so for example, you can just count every ingress packet), but you need to include traffic coming from the fabric interfaces too. So for example: * A single 10Gbps stream coming in one port on the PFE counts as 10Gbps of MQ usage, regardless of whether it is destined for the fabric or for a local port. * A bidirectional 10Gbps stream between two ports on the same PFE counts as 20Gbps of MQ usage, since you have 2 ports each receiving 10Gbps. * 30Gbps of traffic coming in over the fabric (ignoring any fabric limitations for the moment, as they are a separate calculation) and going out the WAN interfaces counts as 30G of MQ usage, which means you still have another 40G available to receive packets from the WAN interfaces and locally or fabric switch them. This is quite a bit more flexible than just thinking about it as 35G full duplex, since you're free to use the 70G in any direction you see fit. -- Richard A Steenbergen r...@e-gerbil.net http://www.e-gerbil.net/ras GPG Key ID: 0xF8B12CBC (7535 7F59 8204 ED1F CC1C 53AF 4C41 5ECA F8B1 2CBC) ___ juniper-nsp mailing list juniper-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/juniper-nsp ___ juniper-nsp mailing list juniper-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/juniper-nsp
Re: [j-nsp] New 16port 10G Card and new MPC with 4x10G MIC Cards - coexistance of old DPCs and new Cards in same chassis -- looking for experience feedback
On Sun, Aug 29, 2010 at 02:29:29AM +0400, Pavel Lunin wrote: My hypotheses is MQ can actually do twice as much: 65 Mpps from the interfaces to back-plane and 65 backwards. Otherwise you'll never get 30 Gbps FD with MPC1. But this knowledge is too burdensome for sales people, because if you don't know it, you can just multiply 65 by the number of chips in a box and get the right pps number. One could hardly understand that each MQ actually does twice as much work but each packet passes two MQ and you need multiply and than divide by 2 accordingly. I got some replies off-list which helped shed some light on the Trio capabilities, so with their permission I will summarize the major points for the archives: * Each Trio PFE is composed of the following ASICs: - MQ: Handles the packet memory, talks to the chassis fabric and the WAN ports, handles port-based QoS, punts first part of the packet to the LU chip for routing lookups. - LU: Lookup ASIC which does all IP routing lookups, MAC lookups, label switching, firewall matching, policing, accounting, etc. - QX: (optional) Implements the fine grained queueing/HQoS stuff. NOT included on the 16-port 10GE MPC. - IX: (optional) Sits in front of the MQ chip to handle GigE ports. * The Trio PFE is good for around 55Mpps of lookups, give or take, depending on the exact operations being performed. * The MQ chip can do around 70Gbps, give or take depending on the packet size. Certain packet sizes can make it all the way to 80Gbps, inconvenient packet sizes can bring it down below 70G by the time you figure in overhead, but the jist is around 70Gbps. This limit is set by the bandwidth of the packet memory. The quoted literature capacity of 60Gbps is intended to be a safe number that can always be met. * The 70G of MQ memory bandwidth is shared between the fabric facing and WAN facing ports, giving you a bidirectional max of 35Gbps each if you run 100% fabric-wan traffic. If you do locally switched wan- wan traffic, you can get the full 70Gbps. On a fabricless chassis like the MX80, that is how you get the entire amount. * The MX960 can only provide around 10Gbps per SCB to each PFE, so it needs to run all 3 SCBs actively to get to 30Gbps. If you lose an SCB, it drops to 20Gbps, etc. This is pre cell overhead, so the actual bandwidth is less (for example, around 28Gbps for 1500 byte packets). * The MX240 and MX480 provide 20Gbps of bandwidth per SCB to each PFE, and will run both actively to get to around 40Gbps (minus the above overhead). Of course the aforementioned 35Gbps memory limit still applies, so even though you have 40Gbps of fabric on these chassis you'll still top out at 35Gbps if you do all fabric-wan traffic. * Anything that is locally switched counts against the LU capacity and the MQ capacity, but not the fabric capacity. As long as you don't exhaust the MQ/fabric, you can get line rate out of the WAN interfaces. For example, 30Gbps of fabric switched + 10Gbps of locally switched traffic on a MX240 or MX480 will not exceed the MQ or fabric capacity and will give you bidirectional line rate. * I'm still hearing mixed information about egress filters affecting local switching, but the latest and most authorative answer is that it DOESN'T actually affect local switching. Everything that can be locally switched supposedly is, including tunnel encapsulation, so if you receive a packet, tunnel it, and send it back out locally, you get 100% free tunneling with no impact to your other capacity. I think that was everything. And if they aren't planning to add it already, please join me in asking them to add a way to view fabric utilization, as it would really make managing the local vs fabric capacities a lot easier. -- Richard A Steenbergen r...@e-gerbil.net http://www.e-gerbil.net/ras GPG Key ID: 0xF8B12CBC (7535 7F59 8204 ED1F CC1C 53AF 4C41 5ECA F8B1 2CBC) ___ juniper-nsp mailing list juniper-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/juniper-nsp
Re: [j-nsp] New 16port 10G Card and new MPC with 4x10G MIC Cards - coexistance of old DPCs and new Cards in same chassis -- looking for experience feedback
Thanks, Richard. 2010/8/29 Richard A Steenbergen r...@e-gerbil.net * Each Trio PFE is composed of the following ASICs: - MQ: Handles the packet memory, talks to the chassis fabric and the WAN ports, handles port-based QoS, punts first part of the packet to the LU chip for routing lookups. - LU: Lookup ASIC which does all IP routing lookups, MAC lookups, label switching, firewall matching, policing, accounting, etc. - QX: (optional) Implements the fine grained queueing/HQoS stuff. NOT included on the 16-port 10GE MPC. - IX: (optional) Sits in front of the MQ chip to handle GigE ports. Here is another joke about '3D' name pronounced by Juniper's people: it's called 3D because it actually consists of four chips. * The Trio PFE is good for around 55Mpps of lookups, give or take, depending on the exact operations being performed. 55, not 65? Anyway, this is what I can't understand (maybe because of my not-native English). When you say 'give or take', you mean it can only do 55/65 for both directions or 55/65 for ethernet-backplane and 55/65 backwards? If this is an LU overall limit for both direction than either I can't manage to convert pps to bps (65pps is about 30 Gigs for 64-byte packets, isn't it) or everything has is twice less performance than we think (not likely though). -- Pavel ___ juniper-nsp mailing list juniper-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/juniper-nsp
Re: [j-nsp] New 16port 10G Card and new MPC with 4x10G MIC Cards - coexistance of old DPCs and new Cards in same chassis -- looking for experience feedback
On Sun, Aug 29, 2010 at 11:00:55AM +0400, Pavel Lunin wrote: * The Trio PFE is good for around 55Mpps of lookups, give or take, depending on the exact operations being performed. 55, not 65? Anyway, this is what I can't understand (maybe because of my not-native English). When you say 'give or take', you mean it can only do 55/65 for both directions or 55/65 for ethernet-backplane and 55/65 backwards? 55 million PACKETS/sec of lookups per PFE, no directionality to that. Lookups are done at ingress. And 55 is what I'm told, not 65. If this is an LU overall limit for both direction than either I can't manage to convert pps to bps (65pps is about 30 Gigs for 64-byte packets, isn't it) or everything has is twice less performance than we think (not likely though). Bandwidth is a completely different limitation, which comes from the MQ and/or fabric limits. Technically 4 ports of 10GE would exceed the 55Mpps, 14.488Mpps * 4 = 57.952Mpps, so you wouldn't quite be able to get line rate on small packets even with local switching. -- Richard A Steenbergen r...@e-gerbil.net http://www.e-gerbil.net/ras GPG Key ID: 0xF8B12CBC (7535 7F59 8204 ED1F CC1C 53AF 4C41 5ECA F8B1 2CBC) ___ juniper-nsp mailing list juniper-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/juniper-nsp
Re: [j-nsp] New 16port 10G Card and new MPC with 4x10G MIC Cards - coexistance of old DPCs and new Cards in same chassis -- looking for experience feedback
Has this always been the case with the SCBs? Will there not be newer SCBs that can run faster? I've always heard that the MX series could potentially run 240gbps per slot but would require SCB upgrade and newer line cards... We're not there yet, but I'm wondering if its true. it sounds like below that we are talking about existing SCBs which means the MX is limited to 120G per slot. From: Richard A Steenbergen r...@e-gerbil.net To: Pavel Lunin plu...@senetsy.ru Cc: juniper-nsp@puck.nether.net juniper-nsp@puck.nether.net Sent: Sun, August 29, 2010 1:39:18 AM Subject: Re: [j-nsp] New 16port 10G Card and new MPC with 4x10G MIC Cards - coexistance of old DPCs and new Cards in same chassis -- looking for experience feedback On Sun, Aug 29, 2010 at 02:29:29AM +0400, Pavel Lunin wrote: My hypotheses is MQ can actually do twice as much: 65 Mpps from the interfaces to back-plane and 65 backwards. Otherwise you'll never get 30 Gbps FD with MPC1. But this knowledge is too burdensome for sales people, because if you don't know it, you can just multiply 65 by the number of chips in a box and get the right pps number. One could hardly understand that each MQ actually does twice as much work but each packet passes two MQ and you need multiply and than divide by 2 accordingly. I got some replies off-list which helped shed some light on the Trio capabilities, so with their permission I will summarize the major points for the archives: * Each Trio PFE is composed of the following ASICs: - MQ: Handles the packet memory, talks to the chassis fabric and the WAN ports, handles port-based QoS, punts first part of the packet to the LU chip for routing lookups. - LU: Lookup ASIC which does all IP routing lookups, MAC lookups, label switching, firewall matching, policing, accounting, etc. - QX: (optional) Implements the fine grained queueing/HQoS stuff. NOT included on the 16-port 10GE MPC. - IX: (optional) Sits in front of the MQ chip to handle GigE ports. * The Trio PFE is good for around 55Mpps of lookups, give or take, depending on the exact operations being performed. * The MQ chip can do around 70Gbps, give or take depending on the packet size. Certain packet sizes can make it all the way to 80Gbps, inconvenient packet sizes can bring it down below 70G by the time you figure in overhead, but the jist is around 70Gbps. This limit is set by the bandwidth of the packet memory. The quoted literature capacity of 60Gbps is intended to be a safe number that can always be met. * The 70G of MQ memory bandwidth is shared between the fabric facing and WAN facing ports, giving you a bidirectional max of 35Gbps each if you run 100% fabric-wan traffic. If you do locally switched wan- wan traffic, you can get the full 70Gbps. On a fabricless chassis like the MX80, that is how you get the entire amount. * The MX960 can only provide around 10Gbps per SCB to each PFE, so it needs to run all 3 SCBs actively to get to 30Gbps. If you lose an SCB, it drops to 20Gbps, etc. This is pre cell overhead, so the actual bandwidth is less (for example, around 28Gbps for 1500 byte packets). * The MX240 and MX480 provide 20Gbps of bandwidth per SCB to each PFE, and will run both actively to get to around 40Gbps (minus the above overhead). Of course the aforementioned 35Gbps memory limit still applies, so even though you have 40Gbps of fabric on these chassis you'll still top out at 35Gbps if you do all fabric-wan traffic. * Anything that is locally switched counts against the LU capacity and the MQ capacity, but not the fabric capacity. As long as you don't exhaust the MQ/fabric, you can get line rate out of the WAN interfaces. For example, 30Gbps of fabric switched + 10Gbps of locally switched traffic on a MX240 or MX480 will not exceed the MQ or fabric capacity and will give you bidirectional line rate. * I'm still hearing mixed information about egress filters affecting local switching, but the latest and most authorative answer is that it DOESN'T actually affect local switching. Everything that can be locally switched supposedly is, including tunnel encapsulation, so if you receive a packet, tunnel it, and send it back out locally, you get 100% free tunneling with no impact to your other capacity. I think that was everything. And if they aren't planning to add it already, please join me in asking them to add a way to view fabric utilization, as it would really make managing the local vs fabric capacities a lot easier. -- Richard A Steenbergen r...@e-gerbil.net http://www.e-gerbil.net/ras GPG Key ID: 0xF8B12CBC (7535 7F59 8204 ED1F CC1C 53AF 4C41 5ECA F8B1 2CBC) ___ juniper-nsp mailing list juniper-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/juniper-nsp ___
Re: [j-nsp] New 16port 10G Card and new MPC with 4x10G MIC Cards - coexistance of old DPCs and new Cards in same chassis -- looking for experience feedback
On Sun, Aug 29, 2010 at 07:03:59AM -0700, Derick Winkworth wrote: Has this always been the case with the SCBs? Will there not be newer SCBs that can run faster? I've always heard that the MX series could potentially run 240gbps per slot but would require SCB upgrade and newer line cards... We're not there yet, but I'm wondering if its true. it sounds like below that we are talking about existing SCBs which means the MX is limited to 120G per slot. Until now each PFE has only needed 10G total bandwidth (per I-chip, * 4 per DPC), so the fabric has been more than sufficient while still providing N+1. My understanding is that even with a new fabric card you'll still be limited to the 35G from the MQ memory bandwidth limit (just like you are with MX240/MX480), so the only difference will be a) you'll get fabric redundancy back, and b) you'll get support for future cards (like 100GE, etc). Another thing I forgot to mention is that the old ADPC I-chip cards can still only talk to the same number of SCB's that they did originally (2x on MX960, 1x on MX240/480). This means that when you're running mixed I-chip and Trio cards in the same chassis, in say for example an MX960, all traffic going to/from an I-chip card will stay on 2 out of 3 SCBs, and only the Trio-to-Trio traffic will be able to use the 3rd SCB. If all of your traffic is going between a Trio card and other I-chip cards, this will obviously bottleneck your Trio capacity at 20G per PFE (minus overhead). Supposedly there is an intelligent fabric request/grant system, so hopefully the Trio PFEs are smart enough to use more capacity on the 3rd SCB for trio-to-trio traffic is the first 2 are being loaded up with I-chip card traffic. You can also use the hidden command show chassis fabric statistics to monitor fabric utilization and drops. The output is pretty difficult to parse, you have to look at it per-plane, and it isn't in XML so you can't even easily write an op script for it, but it's still better than nothing. Hopefully Juniper will add a better fabric utilization command, ideally with something that tracks the peak rate ever seen too (like Cisco does), for example: cisco6509#show platform hardware capacity fabric Switch Fabric Resources Bus utilization: current: 13%, peak was 54% at 08:47:31 UTC Fri Jun 25 2010 Fabric utilization: IngressEgress Module Chanl Speed rate peak rate peak 1 020G1%6% @21:14 06Apr101% 10% @20:14 13Feb10 2 020G 10% 33% @21:15 21Mar100% 31% @20:10 24May10 2 120G2% 52% @03:48 30Apr10 14% 98% @10:20 09Jun10 3 020G 19% 40% @20:38 21Mar10 14% 25% @01:02 09Jul10 3 120G4% 37% @10:42 09Jan101% 61% @02:52 20Dec09 4 020G 27% 51% @20:30 14Jul101%9% @17:04 03May10 4 120G2% 60% @12:12 13May10 34% 82% @01:33 29Apr10 5 020G0%5% @18:51 14Feb100% 21% @18:51 14Feb10 6 020G2% 17% @03:07 29Jun10 19% 52% @17:50 14Jul10 6 120G0% 42% @10:22 20Apr100% 73% @02:25 28Mar10 7 020G6% 33% @10:20 09Jun10 26% 58% @02:25 19Aug10 7 120G 35% 51% @19:38 14Jul101%6% @16:55 03May10 Or at least expose and XML-ify the current one so we can script up something decent. -- Richard A Steenbergen r...@e-gerbil.net http://www.e-gerbil.net/ras GPG Key ID: 0xF8B12CBC (7535 7F59 8204 ED1F CC1C 53AF 4C41 5ECA F8B1 2CBC) ___ juniper-nsp mailing list juniper-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/juniper-nsp
Re: [j-nsp] New 16port 10G Card and new MPC with 4x10G MIC Cards - coexistance of old DPCs and new Cards in same chassis -- looking for experience feedback
so the possibility does exist that with a combination of newer fabric and newer line card (a line card with better MQ memory bandwidth), that MX might be able to push more traffic per slot... From: Richard A Steenbergen r...@e-gerbil.net To: Derick Winkworth dwinkwo...@att.net Cc: juniper-nsp@puck.nether.net juniper-nsp@puck.nether.net Sent: Sun, August 29, 2010 1:34:00 PM Subject: Re: [j-nsp] New 16port 10G Card and new MPC with 4x10G MIC Cards - coexistance of old DPCs and new Cards in same chassis -- looking for experience feedback On Sun, Aug 29, 2010 at 07:03:59AM -0700, Derick Winkworth wrote: Has this always been the case with the SCBs? Will there not be newer SCBs that can run faster? I've always heard that the MX series could potentially run 240gbps per slot but would require SCB upgrade and newer line cards... We're not there yet, but I'm wondering if its true. it sounds like below that we are talking about existing SCBs which means the MX is limited to 120G per slot. Until now each PFE has only needed 10G total bandwidth (per I-chip, * 4 per DPC), so the fabric has been more than sufficient while still providing N+1. My understanding is that even with a new fabric card you'll still be limited to the 35G from the MQ memory bandwidth limit (just like you are with MX240/MX480), so the only difference will be a) you'll get fabric redundancy back, and b) you'll get support for future cards (like 100GE, etc). Another thing I forgot to mention is that the old ADPC I-chip cards can still only talk to the same number of SCB's that they did originally (2x on MX960, 1x on MX240/480). This means that when you're running mixed I-chip and Trio cards in the same chassis, in say for example an MX960, all traffic going to/from an I-chip card will stay on 2 out of 3 SCBs, and only the Trio-to-Trio traffic will be able to use the 3rd SCB. If all of your traffic is going between a Trio card and other I-chip cards, this will obviously bottleneck your Trio capacity at 20G per PFE (minus overhead). Supposedly there is an intelligent fabric request/grant system, so hopefully the Trio PFEs are smart enough to use more capacity on the 3rd SCB for trio-to-trio traffic is the first 2 are being loaded up with I-chip card traffic. You can also use the hidden command show chassis fabric statistics to monitor fabric utilization and drops. The output is pretty difficult to parse, you have to look at it per-plane, and it isn't in XML so you can't even easily write an op script for it, but it's still better than nothing. Hopefully Juniper will add a better fabric utilization command, ideally with something that tracks the peak rate ever seen too (like Cisco does), for example: cisco6509#show platform hardware capacity fabric Switch Fabric Resources Bus utilization: current: 13%, peak was 54% at 08:47:31 UTC Fri Jun 25 2010 Fabric utilization: IngressEgress Module Chanl Speed rate peak rate peak 1 020G1%6% @21:14 06Apr101% 10% @20:14 13Feb10 2 020G 10% 33% @21:15 21Mar100% 31% @20:10 24May10 2 120G2% 52% @03:48 30Apr10 14% 98% @10:20 09Jun10 3 020G 19% 40% @20:38 21Mar10 14% 25% @01:02 09Jul10 3 120G4% 37% @10:42 09Jan101% 61% @02:52 20Dec09 4 020G 27% 51% @20:30 14Jul101%9% @17:04 03May10 4 120G2% 60% @12:12 13May10 34% 82% @01:33 29Apr10 5 020G0%5% @18:51 14Feb100% 21% @18:51 14Feb10 6 020G2% 17% @03:07 29Jun10 19% 52% @17:50 14Jul10 6 120G0% 42% @10:22 20Apr100% 73% @02:25 28Mar10 7 020G6% 33% @10:20 09Jun10 26% 58% @02:25 19Aug10 7 120G 35% 51% @19:38 14Jul101%6% @16:55 03May10 Or at least expose and XML-ify the current one so we can script up something decent. -- Richard A Steenbergen r...@e-gerbil.net http://www.e-gerbil.net/ras GPG Key ID: 0xF8B12CBC (7535 7F59 8204 ED1F CC1C 53AF 4C41 5ECA F8B1 2CBC) ___ juniper-nsp mailing list juniper-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/juniper-nsp
Re: [j-nsp] New 16port 10G Card and new MPC with 4x10G MIC Cards - coexistance of old DPCs and new Cards in same chassis -- looking for experience feedback
Hi Richard, * Supposedly there is some capability to do local switching on the Trio PFE, but I've been told that support is extremely limited, and even doing something like configuring egress filters defeats the local switching and forces everything through the fabric. Sounds as interesting as strange. If this was correct, MX80 would had as limited capabilities as what you described. Moreover I have no idea (do you?) where these limitations can arise. Most stuff like filtering, etc is done by LU, SCB does not do anything with the packet. * Supposedly the reason the MX80 is able to get 60G of performance out of a single Trio PFE is that the normal 30G limitation comes from the MQ ASIC which interconnects Trio components having to split its capacity between MIC-facing and fabric-facing. Because MX80 has no fabric and a single pfe, the MQ can be reconfigured to handle 60G of MIC facing capacity. This would seem to imply that local switching on a MPC could support 30G. This is what I managed to get from local SE. They say a single MQ has 60G overall performance and this is the actual limit. MX80 built-in ports are connected to what is used for fabric-faced interface on MPCs. BTW, this leads to some limitations like impossibility to connect LQ to those 4x10G (per-port queues only). Moreover they also said quite with no doubts (I explicitly asked about this) that it does not matter which direction the traffic goes, you anyway have 60 Gigs per MQ. Say if you only use MX80 biult-in ports, you can have them fully loaded (40 Gbps). Looks like we can assumme the same for 16x10G MPC. More strange thing is why it's written in the datasheet that MX80 (no, I'm not asking anymore why it's MX80 but neither MX120 nor MX60) can do 65 Mpps. 65 Mpps is about 30 Gigs, isn't it? Well, if so, it seems either what I've been told by the SE is a complete nonsense, or someone was drunk preparing the numbers for datasheet. My hypotheses is MQ can actually do twice as much: 65 Mpps from the interfaces to back-plane and 65 backwards. Otherwise you'll never get 30 Gbps FD with MPC1. But this knowledge is too burdensome for sales people, because if you don't know it, you can just multiply 65 by the number of chips in a box and get the right pps number. One could hardly understand that each MQ actually does twice as much work but each packet passes two MQ and you need multiply and than divide by 2 accordingly. -- Pavel ___ juniper-nsp mailing list juniper-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/juniper-nsp
Re: [j-nsp] New 16port 10G Card and new MPC with 4x10G MIC Cards - coexistance of old DPCs and new Cards in same chassis -- looking for experience feedback
* Occasionally the logical subinterface counters will return wildly incorrect data, then mysteriously fix itself on its own after a couple days. We've had difficulty reproducing this one, but I've heard reports of this exact same issue from others doing the same testing, so it can't be THAT big of a fluke. :) I raised a JTAC case on exactly this yesterday for an MX80. At the moment it seems logical units are reporting the overall usage of the entire interface rather than their own. ___ juniper-nsp mailing list juniper-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/juniper-nsp
Re: [j-nsp] New 16port 10G Card and new MPC with 4x10G MIC Cards - coexistance of old DPCs and new Cards in same chassis -- looking for experience feedback
Hi Massimo ! Thx for your advice. We have two reaseons to look at the new MPC Hardware. First we do have the problem, that our Peering MX-960 node slots are nearly full and we need th 16port 10G Card to replace old 4x10G DPCs. Here we have a seconde usecase where we think about using the new MPCs with 4x10G MIC. Our main goal is to use all ports of the card. Thats why we are at a proof of concept LAB in amsterdam at the Juniper site from 2nd to 3rd of September. We want to see if we can break the 120G Limitation by using 2ports per pfe locally without switching over the backplane. In this case the QoS Topic is not relevant as the ISP Node uses iBGP, eBGP and OSPF. So our main interest is in V6 support. The other usecase has to do with MPLS P2MP-LSP for Video distribution. We built a Network for one of our customers for video distribution and used the MX-480 and MX-960 nodes as juniper seemed to support make before break and the ability to add and remove leaves to the P2MP-LSP Tree without packet loss. I guess with binary MCAST replication you mean the binary tree juniper is talking about when it comes to P2MP-LSPs (by the way its CCC we use). Right now we have packet loss in certain situations (depending on network topology) when a leaf is added or removed from an existing P2MP-LSP Tree. Juniper suggests to solve the problem by putting a node with MPC Cards only in the network core to get rid of the problem that a binary tree is always completly recalculated when a leaf is added or removed. So if i understand you right you say that by putting a DPC into a MPC only Node, the MPC Cards notice the DPC and treat P2MP-LSP in the old fashion using the binary tree again. did i understand this correctly ? greetings Gerhard Wir haben neue Telefonnummern und E-Mail Adressen. Bitte aktualisieren Sie den Kontakt. Dipl.-Ing. Gerhard Prochaska Fixed Network Planning Core Development, Implementation Supp A1 Telekom Austria AG Arsenal 22 Obj. 22 A-1030 Wien M: +43 664 66 25690 T: +43 50 664 25690 F: +43 50 664 9 25690 gerhard.procha...@a1telekom.atmailto:gerhard.procha...@a1telekom.at www.A1TelekomAustria.athttp://www.a1telekomaustria.at/ Firmensitz Wien FN: 280571f Handelsgericht Wien P Bitte denken Sie an die Umwelt bevor Sie dieses E-Mail ausdrucken! Von: magno [mailto:massimo.magn...@gmail.com] Gesendet: Freitag, 27. August 2010 01:09 An: Prochaska Gerhard Betreff: Re: [j-nsp] New 16port 10G Card and new MPC with 4x10G MIC Cards - coexistance of old DPCs and new Cards in same chassis -- looking for experience feedback hi Gerhard! I have never used a mixed TRIO - IChip MX, so I can just advice some 'theoretical' concepts to keep in mind: 1) Feature Set: TRIO today is much less rich from a feature set standpoint than I-CHIP. If you need a feature supported on I-CHIP but not on TRIO, you MUST be sure that the whole traffic in both directions that needs that particular feature will be managed on a single PFE (this concept applies also the other way around, for feature supported by TRIO and not by ICHIP). If the traffic needs to be switched throughout the fabric on different kind of DPCs/MPCs only the feature supported by both TRIO and I-CHIP will be supported; 2) Multicast Replication: the enhancement to the binary mcast replication introduced on TRIO won't be available in mixed mode, therefore the legacy binary replication I-CHIP style will be used; Hope this helps to clarify better this topic. Cheers Max. On Thu, Aug 26, 2010 at 1:10 PM, Prochaska Gerhard gerhard.procha...@a1telekom.atmailto:gerhard.procha...@a1telekom.at wrote: Hi ! We are planning to mix DPCs and new Linecards in the same chassis. As far as i understand this is possible from Junos 10.2R2 upwards. We plan to use both types of cards to increase port density on our ISP Nodes where we are short of free slots. So from the Service Point of View iBGP. eBGP, OSPF and Carriers Carrier are the main topics. Im looking for feedback what problems other providers have already encounterd. From reading the posts ist seems that even in an environment with new MPC Cards only Release 10.2p2 is not ready for use in a service provider environment ? Thanks for any feedback. gerhard ___ juniper-nsp mailing list juniper-nsp@puck.nether.netmailto:juniper-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/juniper-nsp ___ juniper-nsp mailing list juniper-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/juniper-nsp
Re: [j-nsp] New 16port 10G Card and new MPC with 4x10G MIC Cards - coexistance of old DPCs and new Cards in same chassis -- looking for experience feedback
On Fri, Aug 27, 2010 at 12:07:53PM +0100, Daniel Goscomb wrote: I raised a JTAC case on exactly this yesterday for an MX80. At the moment it seems logical units are reporting the overall usage of the entire interface rather than their own. We haven't tested vlan tagged interfaces with multiple units yet, our counter bug (which matches what other people have told me they've seen as well) was on an untagged interface with a single .0 logical subint. The physical interface counters appeared to be accurate, but the .0 counters were showing almost no traffic. It could have been showing only locally terminated control plane traffic and not showing interface transit traffic, the numbers would be about right, but we weren't able to confirm that before it went away. -- Richard A Steenbergen r...@e-gerbil.net http://www.e-gerbil.net/ras GPG Key ID: 0xF8B12CBC (7535 7F59 8204 ED1F CC1C 53AF 4C41 5ECA F8B1 2CBC) ___ juniper-nsp mailing list juniper-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/juniper-nsp
Re: [j-nsp] New 16port 10G Card and new MPC with 4x10G MIC Cards - coexistance of old DPCs and new Cards in same chassis -- looking for experience feedback
On Fri, Aug 27, 2010 at 01:45:42PM +0200, Prochaska Gerhard wrote: Our main goal is to use all ports of the card. Thats why we are at a proof of concept LAB in amsterdam at the Juniper site from 2nd to 3rd of September. We want to see if we can break the 120G Limitation by using 2ports per pfe locally without switching over the backplane. I've been unable to get a good explanation of the 120G/slot potential of these cards, or even of the exact performance characteristics under different load conditions. At this point it's more a collection of stories than hard facts, but so far I've heard: * Supposedly there is some capability to do local switching on the Trio PFE, but I've been told that support is extremely limited, and even doing something like configuring egress filters defeats the local switching and forces everything through the fabric. * I've been told that under certain packet conditions, such as 1500 byte packets, you can't even get 30G line rate performance out of each PFE. I haven't been able to get ahold of any details about te math involved, but from what I've heard it's actually better to have smaller packets than bigger ones, the opposite of what you would normally expect to be the limitation. This sounds like it would be a fabric limitation not an LU lookup asic limitation, but who knows. :) * The MX480 is supposedly in a much better fabric situation than the MX960, since it has the same fabric card as the 960 but needs to support fewer slots. Supposedly it can actually deliver something close to 40G of fabric capacity per Trio PFE, where the MX960 can only achieve 30G by running 3 SCBs in active/active/active with no fabric redundancy. * Supposedly the reason the MX80 is able to get 60G of performance out of a single Trio PFE is that the normal 30G limitation comes from the MQ ASIC which interconnects Trio components having to split its capacity between MIC-facing and fabric-facing. Because MX80 has no fabric and a single pfe, the MQ can be reconfigured to handle 60G of MIC facing capacity. This would seem to imply that local switching on a MPC could support 30G. If anybody has better info, I'd absolutely love to hear it. Until then, I'm assuming that the 3D in the MPC card names actually stands for how you'll be using them, 3 ports working, one Disabled. :) -- Richard A Steenbergen r...@e-gerbil.net http://www.e-gerbil.net/ras GPG Key ID: 0xF8B12CBC (7535 7F59 8204 ED1F CC1C 53AF 4C41 5ECA F8B1 2CBC) ___ juniper-nsp mailing list juniper-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/juniper-nsp
Re: [j-nsp] New 16port 10G Card and new MPC with 4x10G MIC Cards - coexistance of old DPCs and new Cards in same chassis -- looking for experience feedback
On Saturday, August 28, 2010 03:49:08 am Richard A Steenbergen wrote: We haven't tested vlan tagged interfaces with multiple units yet, our counter bug (which matches what other people have told me they've seen as well) was on an untagged interface with a single .0 logical subint. The physical interface counters appeared to be accurate, but the .0 counters were showing almost no traffic. It could have been showing only locally terminated control plane traffic and not showing interface transit traffic, the numbers would be about right, but we weren't able to confirm that before it went away. This reminds me of an issue in JUNOS 9.3R2.8 on the EX3200 where SNMP won't tell us about traffic on an interface if the 'description' is applied to the logical unit. Our NOC saw this and figured out the workaround was to move the 'description' to the main interface. Details are sketchy since it's been over a year, but there could have been some weird relationship between this and Cacti for the issue to surface. Mark. signature.asc Description: This is a digitally signed message part. ___ juniper-nsp mailing list juniper-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/juniper-nsp
Re: [j-nsp] New 16port 10G Card and new MPC with 4x10G MIC Cards - coexistance of old DPCs and new Cards in same chassis -- looking for experience feedback
On Sat, Aug 28, 2010 at 09:32:20AM +0800, Mark Tinka wrote: This reminds me of an issue in JUNOS 9.3R2.8 on the EX3200 where SNMP won't tell us about traffic on an interface if the 'description' is applied to the logical unit. Our NOC saw this and figured out the workaround was to move the 'description' to the main interface. Details are sketchy since it's been over a year, but there could have been some weird relationship between this and Cacti for the issue to surface. Hehe funny. I've seen my share of EX counter bugs (we have an awesome one right now where filtered traffic shows up on the subint but not the pysical, still trying to track that one down :P), and actually if you're going all the way back to 9.3 there were plenty of MX counter bugs back then too (double counting, half counting, ae inbound at 0, ae members seeing the entire ae counters, you name it and I saw it between 9.0 and 9.3). But this was definitely just wrong counters, verified with direct SNMP queries. Some people repoted it went away when they restarted mib2d, for us we had to restart mib2d following every RE switchover just to get snmp to return answers period, but it didn't help the logical counter issue. At this point all I can really say is being on the look out for it, and if you do see it make sure Juniper looks at it quickly because it goes away on its own and leave no evidence for them to debug it. :) -- Richard A Steenbergen r...@e-gerbil.net http://www.e-gerbil.net/ras GPG Key ID: 0xF8B12CBC (7535 7F59 8204 ED1F CC1C 53AF 4C41 5ECA F8B1 2CBC) ___ juniper-nsp mailing list juniper-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/juniper-nsp
[j-nsp] New 16port 10G Card and new MPC with 4x10G MIC Cards - coexistance of old DPCs and new Cards in same chassis -- looking for experience feedback
Hi ! We are planning to mix DPCs and new Linecards in the same chassis. As far as i understand this is possible from Junos 10.2R2 upwards. We plan to use both types of cards to increase port density on our ISP Nodes where we are short of free slots. So from the Service Point of View iBGP. eBGP, OSPF and Carriers Carrier are the main topics. Im looking for feedback what problems other providers have already encounterd. From reading the posts ist seems that even in an environment with new MPC Cards only Release 10.2p2 is not ready for use in a service provider environment ? Thanks for any feedback. gerhard ___ juniper-nsp mailing list juniper-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/juniper-nsp
Re: [j-nsp] New 16port 10G Card and new MPC with 4x10G MIC Cards - coexistance of old DPCs and new Cards in same chassis -- looking for experience feedback
On Thu, Aug 26, 2010 at 01:10:37PM +0200, Prochaska Gerhard wrote: Hi ! We are planning to mix DPCs and new Linecards in the same chassis. As far as i understand this is possible from Junos 10.2R2 upwards. We plan to use both types of cards to increase port density on our ISP Nodes where we are short of free slots. So from the Service Point of View iBGP. eBGP, OSPF and Carriers Carrier are the main topics. Im looking for feedback what problems other providers have already encounterd. We have several Trio cards in the earliest stages of deployment, doing some pretty simple carrier-grade IPv4 + IPv6 + MPLS under relatively minor load (probably less than 20Gbps of traffic tops). So far we've seen: * FPCs that crash when you configure certain firewall match criteria and apply it to an interface (time-exceed-bit, for anyone who cares). * SNMP stop responding after RE switchovers, requiring a manual restart of the mib2d process to restore. Was supposed to be fixed in 10.2R2, but we're still seeing it. * Occasionally the logical subinterface counters will return wildly incorrect data, then mysteriously fix itself on its own after a couple days. We've had difficulty reproducing this one, but I've heard reports of this exact same issue from others doing the same testing, so it can't be THAT big of a fluke. :) * When running MPLS with path-mtu allow-fragmentation configured, IPv4-MPLS encapsulating packets will be blackholed. Disable allow-fragmentation as a workaround, should be fixed in 10.2R3. * At one point we saw blackholing of packets across the PFE following an FPC crash, with GRES enabled. Issue went away when you rebooted the backup RE, came back as soon as the backup came back up and tried to start syncing again, cleared up again after turning GRES off. I've seen this issue on and off across several versions of code though, so it might not be Trio specific. As far as reliability goes... I can say that at this exact moment we have no outstanding unresolved issues that are currently impacting our Trio test boxes which can't be worked around... BUT, every time we've turned up a new feature or activated a new service on them we've discovered something new, so I can't vouch that there aren't more issues out there waiting to be discovered. :) Oh and this is a Trio-only deployment, I also can't speak to any additional bugs which will occur when you try to mix ADPC and Trio cards, but I'd be willing to bet money there are some out there. I'm also hoping they will have a service release before 10.2R3, since there are a few other non-Trio related bugs we've found in other code which aren't fixed in 10.2R2. -- Richard A Steenbergen r...@e-gerbil.net http://www.e-gerbil.net/ras GPG Key ID: 0xF8B12CBC (7535 7F59 8204 ED1F CC1C 53AF 4C41 5ECA F8B1 2CBC) ___ juniper-nsp mailing list juniper-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/juniper-nsp
