Re: [c-nsp] Understanding 10G line card oversubscription
I attended a tech briefing that included this information. The release is on target and was told it had been committed to the release. Ie. It is in final QA before release. Your sales engineer can provide more information. The phase 1 release will not support VPLS or EoMPLS. This release is Layer 3 functionality. >From what the briefing provided, this will include almost all of the normal >MPLS functionality (FRR, TE, etc) The phase 2 release is supposed to support Layer 2 over MPLS. The OTV functionality does support Layer 2 and is supposedly better than the equivalent MPLS functionality. Of course this is YATM (Yet Another Tunneling Mechanism) and we will see how it goes. All in all the 7K is starting to get functionality equivalent to the 6500. The new 5k/2k equipment is also pretty nice. The 5548UP is starting to show up on the website with the 5596UP and better 2k FEX are also supposed to be in the works. Check with your SE for data on the new products. Mack McBride Network Architect -Original Message- From: cisco-nsp-boun...@puck.nether.net [mailto:cisco-nsp-boun...@puck.nether.net] On Behalf Of Tim Stevenson Sent: Thursday, March 31, 2011 10:19 AM To: Peter Rathlev; cisco-nsp Subject: Re: [c-nsp] Understanding 10G line card oversubscription It is targeted for the 5.2 release (internal name "Delhi"). Tim At 08:08 AM 3/31/2011, Peter Rathlev uttered: >On Thu, 2011-03-31 at 08:09 -0500, Tony Varriale wrote: > > Phil, looks like Cisco is launching (has launched) their marketing for > > MPLS phase 1 on N7K. > > > > > http://www.cisco.com/en/US/prod/switches/ps9441/ps9402/nexus_7000.html#~MPLS > >Anybody know what software version will support MPLS on N7k? I can't >seem to see that from the above. > >-- >Peter > > >___ >cisco-nsp mailing list cisco-nsp@puck.nether.net >https://puck.nether.net/mailman/listinfo/cisco-nsp >archive at http://puck.nether.net/pipermail/cisco-nsp/ Tim Stevenson, tstev...@cisco.com Routing & Switching CCIE #5561 Distinguished Technical Marketing Engineer, Cisco Nexus 7000 Cisco - http://www.cisco.com IP Phone: 408-526-6759 The contents of this message may be *Cisco Confidential* and are intended for the specified recipients only. ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/ ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/
Re: [c-nsp] Understanding 10G line card oversubscription
It is targeted for the 5.2 release (internal name "Delhi"). Tim At 08:08 AM 3/31/2011, Peter Rathlev uttered: On Thu, 2011-03-31 at 08:09 -0500, Tony Varriale wrote: > Phil, looks like Cisco is launching (has launched) their marketing for > MPLS phase 1 on N7K. > > http://www.cisco.com/en/US/prod/switches/ps9441/ps9402/nexus_7000.html#~MPLS Anybody know what software version will support MPLS on N7k? I can't seem to see that from the above. -- Peter ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/ Tim Stevenson, tstev...@cisco.com Routing & Switching CCIE #5561 Distinguished Technical Marketing Engineer, Cisco Nexus 7000 Cisco - http://www.cisco.com IP Phone: 408-526-6759 The contents of this message may be *Cisco Confidential* and are intended for the specified recipients only. ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/
Re: [c-nsp] Understanding 10G line card oversubscription
On Thu, 2011-03-31 at 08:09 -0500, Tony Varriale wrote: > Phil, looks like Cisco is launching (has launched) their marketing for > MPLS phase 1 on N7K. > > http://www.cisco.com/en/US/prod/switches/ps9441/ps9402/nexus_7000.html#~MPLS Anybody know what software version will support MPLS on N7k? I can't seem to see that from the above. -- Peter ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/
Re: [c-nsp] Understanding 10G line card oversubscription
On 3/23/2011 3:57 AM, Phil Mayers wrote: The N7k is a nice platform in many ways. Far higher performance, better software and some interesting features like mcLAG. It would be a great fit for us, *if* it had the MPLS feature set. It doesn't == a shame (for us) Phil, looks like Cisco is launching (has launched) their marketing for MPLS phase 1 on N7K. http://www.cisco.com/en/US/prod/switches/ps9441/ps9402/nexus_7000.html#~MPLS tv ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/
Re: [c-nsp] Understanding 10G line card oversubscription
On 3/23/2011 3:57 AM, Phil Mayers wrote: Why would I bother listening to details/timelines from them? They've been wildly, wildly inaccurate in the past. True. Some things (like Sup2T) are worse than others. At this point, Cisco could tell me "it's out next week" and I wouldn't base purchasing decisions on it ;o) LOL! Yeah I wouldn't do that either. The N7k is a nice platform in many ways. Far higher performance, better software and some interesting features like mcLAG. It would be a great fit for us, *if* it had the MPLS feature set. It doesn't == a shame (for us) Some people are more plugged into the beta and release process for the Nexus line. I've been fortunately enough to do that for a number of features across N7K, N5K and the N1010. When those features were solid during the beta phase, they were released appropriately and fairly quickly. So, that's all I can say right now :) What do you expect me to do? Give them a round of applause? They do get *paid* to do this after all. And let's face it - "fairly good job" is not exactly a ringing endorsement. Congratulations! You're not doing badly! ;o) LOL! No. But, I would say given the past with the 6500, I much prefer the N7K and think it's been a much more positive process. Bug free? Absolutely not. At the end of the day, I'm sure the nexus range will sell fine without weirdo edge-case UK universities as a customer. But for me, it's a shame that, where the 6500 was a swiss army knife, the new platforms seem to be less so. It worked well for us, giving us great performance at relatively low cost and ease of sparing, testing and deployment. I'm not sure that edge N7K deployments are weird. But, if you are interested I'm always willing to listen/learn about things people are doing with their equipment (or where new ones may fit). Feel free to contact me off list if you want to chat. tv ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/
Re: [c-nsp] Understanding 10G line card oversubscription
On 03/22/2011 08:23 PM, Tony Varriale wrote: I've heard "very shortly" from Cisco before. Frankly, they've got no belief credits with me. Unless and until I see it, it's vapour. We all have. If you are considering the platform and need those features, get a hold of your partner and/or Cisco account team. Unfortunately I can't share those details/timelines. Why would I bother listening to details/timelines from them? They've been wildly, wildly inaccurate in the past. At this point, Cisco could tell me "it's out next week" and I wouldn't base purchasing decisions on it ;o) Considering it's a 3 year old platform, you are asking for a lot IMO. The n7k wasn't really meant for what's it's doing and going to do. Oh, the 6500 platform is 11 years old this year. But, there are other platforms that meet your requirements. Sure. Other vendors, too... Which is a shame, because a lot of the Nexus features look great; NX-OS certainly seems to have a better, newer structure and both the control and forwarding plane are a lot faster on the N7k. I'm not sure I get the shame part. Which part is a shame? The N7k is a nice platform in many ways. Far higher performance, better software and some interesting features like mcLAG. It would be a great fit for us, *if* it had the MPLS feature set. It doesn't == a shame (for us) I would love to be proven wrong. Maybe NX-OS is built in such a way as to permit speedy feature development. But unless and until it's shipping, and the bugs are ironed out... Any software development cycle is going to be that. As stated above, Nexus wasn't really expecting to support service modules, MPLS other whiz bang features initially. But, it will. So, I think they are doing a fairly good job considering. What do you expect me to do? Give them a round of applause? They do get *paid* to do this after all. And let's face it - "fairly good job" is not exactly a ringing endorsement. Congratulations! You're not doing badly! ;o) At the end of the day, I'm sure the nexus range will sell fine without weirdo edge-case UK universities as a customer. But for me, it's a shame that, where the 6500 was a swiss army knife, the new platforms seem to be less so. It worked well for us, giving us great performance at relatively low cost and ease of sparing, testing and deployment. Others I'm sure disagree and that's fine - but I can only speak for myself. ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/
Re: [c-nsp] Understanding 10G line card oversubscription
The connection of the port pairs on the 6708 is limited to 16GB each direction to the fabric asic. On the 6716 the groups are four ports which share a connection of 10GB each direction to the fabric asic. Mack -Original Message- From: John Neiberger [mailto:jneiber...@gmail.com] Sent: Tuesday, March 22, 2011 10:56 AM To: Mack McBride Cc: Sergey Nikitin; D B; cisco-nsp@puck.nether.net Subject: Re: [c-nsp] Understanding 10G line card oversubscription I was under the impression that the total bandwidth available per four-port group was 40G, so the limiting factor on the 6708 was backplane connectivity (2x20G), but perhaps that has more to do with multicast replication capacity. What is this 16G limit you're referring to? John On Tue, Mar 22, 2011 at 10:28 AM, Mack McBride wrote: > I had forgotten about that break out session but it does correspond with what > I thought. > The over-subscription per four port group is 10G on the 6716 as opposed to > 16G on the 6708. > > Mack > > -Original Message- > From: cisco-nsp-boun...@puck.nether.net > [mailto:cisco-nsp-boun...@puck.nether.net] On Behalf Of Sergey Nikitin > Sent: Tuesday, March 22, 2011 1:17 AM > To: D B > Cc: cisco-nsp@puck.nether.net > Subject: Re: [c-nsp] Understanding 10G line card oversubscription > > Try to look at this: > > http://depositfiles.com/ru/files/j2b46xu52 > > or google BRKARC-3465, page 33/34. > > D B wrote: >> I'm looking for in-depth documentation on switch fabric oversubscription for >> these two 10G line cards: >> WS-X6708-10GE >> WS-X6716-10GE >> >> I'd like to understand its design and operation regarding these cards. Also, >> how to identify/quantify instances where oversubscription is nearing/hitting >> thresholds that will cause packet drops (SNMP?). >> ___ >> cisco-nsp mailing list cisco-nsp@puck.nether.net >> https://puck.nether.net/mailman/listinfo/cisco-nsp >> archive at http://puck.nether.net/pipermail/cisco-nsp/ >> > ___ > cisco-nsp mailing list cisco-nsp@puck.nether.net > https://puck.nether.net/mailman/listinfo/cisco-nsp > archive at http://puck.nether.net/pipermail/cisco-nsp/ > > ___ > cisco-nsp mailing list cisco-nsp@puck.nether.net > https://puck.nether.net/mailman/listinfo/cisco-nsp > archive at http://puck.nether.net/pipermail/cisco-nsp/ > ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/
Re: [c-nsp] Understanding 10G line card oversubscription
I've heard "very shortly" from Cisco before. Frankly, they've got no belief credits with me. Unless and until I see it, it's vapour. We all have. If you are considering the platform and need those features, get a hold of your partner and/or Cisco account team. Unfortunately I can't share those details/timelines. As for features - full parity with the feature set on 6500, so: L3VPN including 6vPE EoMPLS FRR Autotunnel And although this isn't on 6500, on a newer platform I expect VPLS for good measure. I heard that the feature will be there. Now, the chipset inside the N7k may be capable of a bunch of wondrous things, but without the software it's just so much expensive fused sand. I find it *extraordinarily* hard to believe they'll reach MPLS feature parity with the (current) 6500 in any timescale less than 2 years. Considering it's a 3 year old platform, you are asking for a lot IMO. The n7k wasn't really meant for what's it's doing and going to do. Oh, the 6500 platform is 11 years old this year. But, there are other platforms that meet your requirements. Which is a shame, because a lot of the Nexus features look great; NX-OS certainly seems to have a better, newer structure and both the control and forwarding plane are a lot faster on the N7k. I'm not sure I get the shame part. Which part is a shame? I would love to be proven wrong. Maybe NX-OS is built in such a way as to permit speedy feature development. But unless and until it's shipping, and the bugs are ironed out... Any software development cycle is going to be that. As stated above, Nexus wasn't really expecting to support service modules, MPLS other whiz bang features initially. But, it will. So, I think they are doing a fairly good job considering. tv ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/
Re: [c-nsp] Understanding 10G line card oversubscription
I was under the impression that the total bandwidth available per four-port group was 40G, so the limiting factor on the 6708 was backplane connectivity (2x20G), but perhaps that has more to do with multicast replication capacity. What is this 16G limit you're referring to? John On Tue, Mar 22, 2011 at 10:28 AM, Mack McBride wrote: > I had forgotten about that break out session but it does correspond with what > I thought. > The over-subscription per four port group is 10G on the 6716 as opposed to > 16G on the 6708. > > Mack > > -Original Message- > From: cisco-nsp-boun...@puck.nether.net > [mailto:cisco-nsp-boun...@puck.nether.net] On Behalf Of Sergey Nikitin > Sent: Tuesday, March 22, 2011 1:17 AM > To: D B > Cc: cisco-nsp@puck.nether.net > Subject: Re: [c-nsp] Understanding 10G line card oversubscription > > Try to look at this: > > http://depositfiles.com/ru/files/j2b46xu52 > > or google BRKARC-3465, page 33/34. > > D B wrote: >> I'm looking for in-depth documentation on switch fabric oversubscription for >> these two 10G line cards: >> WS-X6708-10GE >> WS-X6716-10GE >> >> I'd like to understand its design and operation regarding these cards. Also, >> how to identify/quantify instances where oversubscription is nearing/hitting >> thresholds that will cause packet drops (SNMP?). >> ___ >> cisco-nsp mailing list cisco-nsp@puck.nether.net >> https://puck.nether.net/mailman/listinfo/cisco-nsp >> archive at http://puck.nether.net/pipermail/cisco-nsp/ >> > ___ > cisco-nsp mailing list cisco-nsp@puck.nether.net > https://puck.nether.net/mailman/listinfo/cisco-nsp > archive at http://puck.nether.net/pipermail/cisco-nsp/ > > ___ > cisco-nsp mailing list cisco-nsp@puck.nether.net > https://puck.nether.net/mailman/listinfo/cisco-nsp > archive at http://puck.nether.net/pipermail/cisco-nsp/ > ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/
Re: [c-nsp] Understanding 10G line card oversubscription
I had forgotten about that break out session but it does correspond with what I thought. The over-subscription per four port group is 10G on the 6716 as opposed to 16G on the 6708. Mack -Original Message- From: cisco-nsp-boun...@puck.nether.net [mailto:cisco-nsp-boun...@puck.nether.net] On Behalf Of Sergey Nikitin Sent: Tuesday, March 22, 2011 1:17 AM To: D B Cc: cisco-nsp@puck.nether.net Subject: Re: [c-nsp] Understanding 10G line card oversubscription Try to look at this: http://depositfiles.com/ru/files/j2b46xu52 or google BRKARC-3465, page 33/34. D B wrote: > I'm looking for in-depth documentation on switch fabric oversubscription for > these two 10G line cards: > WS-X6708-10GE > WS-X6716-10GE > > I'd like to understand its design and operation regarding these cards. Also, > how to identify/quantify instances where oversubscription is nearing/hitting > thresholds that will cause packet drops (SNMP?). > ___ > cisco-nsp mailing list cisco-nsp@puck.nether.net > https://puck.nether.net/mailman/listinfo/cisco-nsp > archive at http://puck.nether.net/pipermail/cisco-nsp/ > ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/ ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/
Re: [c-nsp] Understanding 10G line card oversubscription
I hadn't seen this linked anywhere in the thread - but this is probably what you were look for. http://www.cisco.com/en/US/prod/collateral/switches/ps5718/ps708/prod_white_paper0900aecd80673385.html Cheers Andrew ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/
Re: [c-nsp] Understanding 10G line card oversubscription
Hi, On Tue, Mar 22, 2011 at 08:57:59AM +, Phil Mayers wrote: > Especially so if you insist on letting bits of your company compete > against each other with subtly different platforms ;o) > > I had a long reply here, but basically: Cisco made their choice. They > chose to go "clean slate" and discard their own experience. I'm sure > they'll work their way back up the learning curve. When we come to > re-procure, we'll see how they do - it's always an interesting experience. I find this actually not all bad... On one side, the 6500/7600 split and subsequent customer annoyance was one of the most stupid thing they could have ever thought up - same hardware, same software (-architecture), but arbitrary hurdles just to be different. Just imagine how powerful the 6500/7600 platform could have been if they had joined forces, instead of competing internally, and spending twice the development effort (Sup720 and RSP720, new -E chassis and -S chassis, etc.) OTOH, completely ditching IOS and going for a new and clean software architecture (and I'm not sure I consider XE "clean" for any definition of the word, while it's indeed a "new architecture"...) is something I strongly applaud the Nexus team for. What I'm worried about is that Cisco is again making the mistake of splitting engineering resources (classic IOS, XR, XE, NX-OS) and so all the "new" OSes are still lacking features, are competing with each other, instead of having a single decent OS for their router and switch plattforms. gert -- USENET is *not* the non-clickable part of WWW! //www.muc.de/~gert/ Gert Doering - Munich, Germany g...@greenie.muc.de fax: +49-89-35655025g...@net.informatik.tu-muenchen.de pgp8JjnF1hd5f.pgp Description: PGP signature ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/
Re: [c-nsp] Understanding 10G line card oversubscription
On 03/22/2011 12:50 AM, Nick Hilliard wrote: On 21/03/2011 22:22, Phil Mayers wrote: Having said that, the truly tragic thing about the 6500 is that, until recently, it still beat a lot of newer platforms on feature mix combined with decent performance and reasonable (if not great) density. [...] Having said that, I won't be sorry to see the back of the crappy CPU and 12.2S IOS train ;o) that works both ways. Because the switch has been around for many years and has been such a cash-cow for Cisco, it's been financially possible for cisco to fund development of a very large number of features on the system. If you start out from scratch with a new pile of silicon (asr9k / n7k or indeed any other product line), you end up spending huge quantities of money even to get close to feature parity to existing products, by which time your hardware is outdated. It's a really difficult problem to deal with. Especially so if you insist on letting bits of your company compete against each other with subtly different platforms ;o) I had a long reply here, but basically: Cisco made their choice. They chose to go "clean slate" and discard their own experience. I'm sure they'll work their way back up the learning curve. When we come to re-procure, we'll see how they do - it's always an interesting experience. ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/
Re: [c-nsp] Understanding 10G line card oversubscription
On 03/22/2011 01:03 AM, Tony Varriale wrote: The Nexus 7K is working on the routing side of things but lacks features (MPLS). I suspect that's coming very shortly. Which MPLS features are important to you? I've heard "very shortly" from Cisco before. Frankly, they've got no belief credits with me. Unless and until I see it, it's vapour. As for features - full parity with the feature set on 6500, so: L3VPN including 6vPE EoMPLS FRR Autotunnel And although this isn't on 6500, on a newer platform I expect VPLS for good measure. Now, the chipset inside the N7k may be capable of a bunch of wondrous things, but without the software it's just so much expensive fused sand. I find it *extraordinarily* hard to believe they'll reach MPLS feature parity with the (current) 6500 in any timescale less than 2 years. Which is a shame, because a lot of the Nexus features look great; NX-OS certainly seems to have a better, newer structure and both the control and forwarding plane are a lot faster on the N7k. I would love to be proven wrong. Maybe NX-OS is built in such a way as to permit speedy feature development. But unless and until it's shipping, and the bugs are ironed out... ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/
Re: [c-nsp] Understanding 10G line card oversubscription
Try to look at this: http://depositfiles.com/ru/files/j2b46xu52 or google BRKARC-3465, page 33/34. D B wrote: I'm looking for in-depth documentation on switch fabric oversubscription for these two 10G line cards: WS-X6708-10GE WS-X6716-10GE I'd like to understand its design and operation regarding these cards. Also, how to identify/quantify instances where oversubscription is nearing/hitting thresholds that will cause packet drops (SNMP?). ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/ ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/
Re: [c-nsp] Understanding 10G line card oversubscription
It's entirely possible that we just have a very weird mix of requirements... Care to share a couple of them? Having said that, I won't be sorry to see the back of the crappy CPU and 12.2S IOS train ;o) Can I add to your list: eFSU, OIR, fabs and sups living together and punting to CPU on 3rd down? :) tv ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/
Re: [c-nsp] Understanding 10G line card oversubscription
On 3/21/2011 6:33 PM, Mack McBride wrote: The 6500 is still quite good if you don't have high throughput requirements (<80G). Between that and the many times delay of the Sup2T, Nexus is a $1B business now. The newer Cisco platforms don't do full routing and switching well. Which ones? The Nexus 7K is working on the routing side of things but lacks features (MPLS). I suspect that's coming very shortly. Which MPLS features are important to you? tv ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/
Re: [c-nsp] Understanding 10G line card oversubscription
On 21/03/2011 22:22, Phil Mayers wrote: Having said that, the truly tragic thing about the 6500 is that, until recently, it still beat a lot of newer platforms on feature mix combined with decent performance and reasonable (if not great) density. [...] Having said that, I won't be sorry to see the back of the crappy CPU and 12.2S IOS train ;o) that works both ways. Because the switch has been around for many years and has been such a cash-cow for Cisco, it's been financially possible for cisco to fund development of a very large number of features on the system. If you start out from scratch with a new pile of silicon (asr9k / n7k or indeed any other product line), you end up spending huge quantities of money even to get close to feature parity to existing products, by which time your hardware is outdated. It's a really difficult problem to deal with. Nick ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/
Re: [c-nsp] Understanding 10G line card oversubscription
I have to agree with feature mix/performance/cost in a single platform. The 6500 is still quite good if you don't have high throughput requirements (<80G). The newer Cisco platforms don't do full routing and switching well. The Nexus 7K is working on the routing side of things but lacks features (MPLS). The ASR 9K does some switching but lacks the familiar R-PVST+ (not that I don't like MST but there is a learning curve). Your mileage with other vendors may vary. Mack -Original Message- From: cisco-nsp-boun...@puck.nether.net [mailto:cisco-nsp-boun...@puck.nether.net] On Behalf Of Phil Mayers Sent: Monday, March 21, 2011 4:22 PM To: cisco-nsp@puck.nether.net Subject: Re: [c-nsp] Understanding 10G line card oversubscription On 03/21/2011 09:41 PM, Greg Whynott wrote: > the 6500's are not very well suited for many roles these days in DC land, > especially if your into HPC.if you are using a policy engine such as the > FWSM, you now only have 4G if your traffic has to pass threw it. if memory > serves me correctly. We started seeing input drops which lead us to start > a deployment OSPF-ECMP so we could balance and route around the core 6500.. > > kinda sad when you consider the cost of the kit.we bought a pair of > extreme switches last round, they blow the socks of anything we have had > from Cisco (so far!) in terms of features, performance and cost of ownership. > > just my opinion, i may be way off base. Well, I think it depends on what you're doing. It is certainly a poor platform for high-density 10gig. Many caveats and cost is way, way too high. Transceiver tedium too :o( Having said that, the truly tragic thing about the 6500 is that, until recently, it still beat a lot of newer platforms on feature mix combined with decent performance and reasonable (if not great) density. It's entirely possible that we just have a very weird mix of requirements, but I'm honestly not looking forward to replacing our 6500s - it's not obvious to me we can get the same feature mix, better performance and good cost in a single device, which means a re-architect, as well as higher cost in carrying spares :o( Having said that, I won't be sorry to see the back of the crappy CPU and 12.2S IOS train ;o) ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/ ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/
Re: [c-nsp] Understanding 10G line card oversubscription
Just guessing but it would appear that the NAXOS act as two port asics. These feed data to the R2D2/ASHWINI pair. My guess is the ASHWINI are between the NAXOS and R2D2 chip since we know >From the 6704 that the R2D2 can interface directly with the Metro. Since the R2D2 has a 10G interface, I think it is safe to say the 6716 only has 10G for each 4 port group as Tim stated. So from a strictly port group over-subscription perspective the 6716 is a worse blade than the 6708. Mack -Original Message- From: Phil Mayers [mailto:p.may...@imperial.ac.uk] Sent: Monday, March 21, 2011 4:26 PM To: Mack McBride Cc: cisco-nsp@puck.nether.net Subject: Re: [c-nsp] Understanding 10G line card oversubscription On 03/21/2011 09:26 PM, Mack McBride wrote: > There is the 4 port group which I believe is the same as a port pair on a > 6708. Ie. No free switching. > Based on some comments by Tim Stevenson, the link between the four port group > and the fabric asic is 10G > rather than the 16G in the 6708. But he mentions some of the same chips > (metro, r2d2) so the architecture > can't be much different from the 6708. > > Then there is the fabric group which is switched locally without going over > the fabric. > > show asic slot Module in slot 1 has 7 type(s) of ASICs ASIC Name Count Version KUMA 2 (3.0) METRO_ARGOS 2 (3.0) METRO_KRYPTON 2 (3.0) SSA 2 (9.0) NAXOS 8 (1.0) R2D2 4 (3.0) ASHWINI 4 (0.6) > sh int capabilities | inc ASIC TenGigabitEthernet1/1 Ports-in-ASIC (Sub-port ASIC) : 1-8 (1-4) TenGigabitEthernet1/5 Ports-in-ASIC (Sub-port ASIC) : 1-8 (5-8) TenGigabitEthernet1/9 Ports-in-ASIC (Sub-port ASIC) : 9-16 (9-12) TenGigabitEthernet1/13 Ports-in-ASIC (Sub-port ASIC) : 9-16 (13-16) > Short answer seems to be that the 6716 is basically the same despite claims > to the contrary > Hmm. I wonder if anyone from Cisco will comment ;o) ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/
Re: [c-nsp] Understanding 10G line card oversubscription
On 03/21/2011 09:26 PM, Mack McBride wrote: There is the 4 port group which I believe is the same as a port pair on a 6708. Ie. No free switching. Based on some comments by Tim Stevenson, the link between the four port group and the fabric asic is 10G rather than the 16G in the 6708. But he mentions some of the same chips (metro, r2d2) so the architecture can't be much different from the 6708. Then there is the fabric group which is switched locally without going over the fabric. show asic slot Module in slot 1 has 7 type(s) of ASICs ASIC Name Count Version KUMA 2 (3.0) METRO_ARGOS 2 (3.0) METRO_KRYPTON 2 (3.0) SSA 2 (9.0) NAXOS 8 (1.0) R2D2 4 (3.0) ASHWINI 4 (0.6) sh int capabilities | inc ASIC TenGigabitEthernet1/1 Ports-in-ASIC (Sub-port ASIC) : 1-8 (1-4) TenGigabitEthernet1/5 Ports-in-ASIC (Sub-port ASIC) : 1-8 (5-8) TenGigabitEthernet1/9 Ports-in-ASIC (Sub-port ASIC) : 9-16 (9-12) TenGigabitEthernet1/13 Ports-in-ASIC (Sub-port ASIC) : 9-16 (13-16) Short answer seems to be that the 6716 is basically the same despite claims to the contrary Hmm. I wonder if anyone from Cisco will comment ;o) ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/
Re: [c-nsp] Understanding 10G line card oversubscription
On 03/21/2011 09:41 PM, Greg Whynott wrote: the 6500's are not very well suited for many roles these days in DC land, especially if your into HPC.if you are using a policy engine such as the FWSM, you now only have 4G if your traffic has to pass threw it. if memory serves me correctly. We started seeing input drops which lead us to start a deployment OSPF-ECMP so we could balance and route around the core 6500.. kinda sad when you consider the cost of the kit.we bought a pair of extreme switches last round, they blow the socks of anything we have had from Cisco (so far!) in terms of features, performance and cost of ownership. just my opinion, i may be way off base. Well, I think it depends on what you're doing. It is certainly a poor platform for high-density 10gig. Many caveats and cost is way, way too high. Transceiver tedium too :o( Having said that, the truly tragic thing about the 6500 is that, until recently, it still beat a lot of newer platforms on feature mix combined with decent performance and reasonable (if not great) density. It's entirely possible that we just have a very weird mix of requirements, but I'm honestly not looking forward to replacing our 6500s - it's not obvious to me we can get the same feature mix, better performance and good cost in a single device, which means a re-architect, as well as higher cost in carrying spares :o( Having said that, I won't be sorry to see the back of the crappy CPU and 12.2S IOS train ;o) ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/
Re: [c-nsp] Understanding 10G line card oversubscription
The 6500 is a fairly old platform but it will be around for a while yet. Especially if they every release a faster supervisor/fabric. The 6500 really made Cisco a market leader. Mack -Original Message- From: cisco-nsp-boun...@puck.nether.net [mailto:cisco-nsp-boun...@puck.nether.net] On Behalf Of Scott Granados Sent: Monday, March 21, 2011 4:03 PM To: Nick Hilliard Cc: cisco-nsp@puck.nether.net Service Providers Subject: Re: [c-nsp] Understanding 10G line card oversubscription When you consider the age of the 65xx platform it's pretty remarkable how they have held up. On Mar 21, 2011, at 2:55 PM, Nick Hilliard wrote: > On 21/03/2011 21:41, Greg Whynott wrote: >> just my opinion, i may be way off base. > > no, not at all - it's not surprising that something with a chipset from 2 > years ago will spank it. The 6500 is a great GE aggregator switch, but > only a mediocre 10G platform. Retro-fitting high density 10G on a > backplane dating from the early 2000s is not really a viable option if you > actually need to push lots of data through it, so it's not surprising that > it's no good for HPC. > > If you can live within their limitations, the current generation of 10G > top-of-rack switches provide ridiculously good value. > > Nick > ___ > cisco-nsp mailing list cisco-nsp@puck.nether.net > https://puck.nether.net/mailman/listinfo/cisco-nsp > archive at http://puck.nether.net/pipermail/cisco-nsp/ ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/ ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/
Re: [c-nsp] Understanding 10G line card oversubscription
When you consider the age of the 65xx platform it's pretty remarkable how they have held up. On Mar 21, 2011, at 2:55 PM, Nick Hilliard wrote: > On 21/03/2011 21:41, Greg Whynott wrote: >> just my opinion, i may be way off base. > > no, not at all - it's not surprising that something with a chipset from 2 > years ago will spank it. The 6500 is a great GE aggregator switch, but > only a mediocre 10G platform. Retro-fitting high density 10G on a > backplane dating from the early 2000s is not really a viable option if you > actually need to push lots of data through it, so it's not surprising that > it's no good for HPC. > > If you can live within their limitations, the current generation of 10G > top-of-rack switches provide ridiculously good value. > > Nick > ___ > cisco-nsp mailing list cisco-nsp@puck.nether.net > https://puck.nether.net/mailman/listinfo/cisco-nsp > archive at http://puck.nether.net/pipermail/cisco-nsp/ ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/
Re: [c-nsp] Understanding 10G line card oversubscription
On 21/03/2011 21:41, Greg Whynott wrote: just my opinion, i may be way off base. no, not at all - it's not surprising that something with a chipset from 2 years ago will spank it. The 6500 is a great GE aggregator switch, but only a mediocre 10G platform. Retro-fitting high density 10G on a backplane dating from the early 2000s is not really a viable option if you actually need to push lots of data through it, so it's not surprising that it's no good for HPC. If you can live within their limitations, the current generation of 10G top-of-rack switches provide ridiculously good value. Nick ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/
Re: [c-nsp] Understanding 10G line card oversubscription
the 6500's are not very well suited for many roles these days in DC land, especially if your into HPC.if you are using a policy engine such as the FWSM, you now only have 4G if your traffic has to pass threw it. if memory serves me correctly. We started seeing input drops which lead us to start a deployment OSPF-ECMP so we could balance and route around the core 6500.. kinda sad when you consider the cost of the kit.we bought a pair of extreme switches last round, they blow the socks of anything we have had from Cisco (so far!) in terms of features, performance and cost of ownership. just my opinion, i may be way off base. greg On Mar 21, 2011, at 5:26 PM, Mack McBride wrote: > There is the 4 port group which I believe is the same as a port pair on a > 6708. Ie. No free switching. > Based on some comments by Tim Stevenson, the link between the four port group > and the fabric asic is 10G > rather than the 16G in the 6708. But he mentions some of the same chips > (metro, r2d2) so the architecture > can't be much different from the 6708. > > Then there is the fabric group which is switched locally without going over > the fabric. > > show asic slot > sh int capabilities | inc ASIC > > Of course a diagram would be really helpful but those usually come with an > NDA. > > 6708 results: > >ASIC Name Count Version > KUMA 2 (3.0) > METRO_ARGOS 2 (3.0) >METRO_KRYPTON 2 (3.0) > SSA 2 (9.0) > R2D2 8 (2.0) > TIANGANG 4 (54.0) > > Te/1 > Ports-in-ASIC (Sub-port ASIC) : 1,4-5,7 (1) > Te/2 > Ports-in-ASIC (Sub-port ASIC) : 2-3,6,8 (2) > > My understanding of the chips is as follows: > > R2D2(per port) -> Tiangang (port pair) -> Metro argos -> SSA (Fabric complex) > Metro Krypto interfaces to the EARL complex aka PFC (local switching/routing > to avoid fabric) > Kuma acts as a bus bridge (not relevant to fabric switching) > > The assumption is that the Tiangang is replace with another chip that can > handle > 4 ports. I assume the new FPGA - Metro link is somehow different on the 6716 > but > I don't know how. > > The 6704 skips the tiangang chips and directly connects the R2D2 to the Metro. > > From a Cisco doc > (http://www.cisco.com/en/US/docs/ios/vswitch/command/reference/vs_02.pdf): >ASIC Name Count Version > KUMA 2 (2.0) > METRO_ARGOS 2 (2.0) > METRO_KRYPTON 2 (2.0) > SSA 2 (8.0) > R2D2 4 (2.0) > > Short answer seems to be that the 6716 is basically the same despite claims > to the contrary > > Mack > > -Original Message----- > From: cisco-nsp-boun...@puck.nether.net > [mailto:cisco-nsp-boun...@puck.nether.net] On Behalf Of Phil Mayers > Sent: Monday, March 21, 2011 1:51 PM > To: cisco-nsp@puck.nether.net > Subject: Re: [c-nsp] Understanding 10G line card oversubscription > > On 03/21/2011 06:33 PM, Mack McBride wrote: > >> The 6716 is going to have similar limitations but I don't have a good >> document on how the port asics connect > > My understanding is that the 6716 is quite different from the 6708. > There's no "free" local switching within port groups AFAIK. The > differences have been discussed on the list before. > >> >> If someone can verify the connection method and limitations on the 6716 it >> would be appreciated. > > Do you have specific IOS commands? We've got a few in service. > ___ > cisco-nsp mailing list cisco-nsp@puck.nether.net > https://puck.nether.net/mailman/listinfo/cisco-nsp > archive at http://puck.nether.net/pipermail/cisco-nsp/ > > ___ > cisco-nsp mailing list cisco-nsp@puck.nether.net > https://puck.nether.net/mailman/listinfo/cisco-nsp > archive at http://puck.nether.net/pipermail/cisco-nsp/ Gregory Whynott Networks and Storage Ontario Institute for Cancer Research MaRS Centre, South Tower 101 College Street, Suite 800 Toronto, Ontario, Canada M5G 0A3 647-294-2813 | www.oicr.on.ca -- This message and any attachments may contain confidential and/or privileged information for the sole use of the intended recipient. Any review or distribution by anyone other than the person for whom it was originally intended is strictly prohibited. If you have received this message in error, please contact the sender and delete all copies. Opinions, conclusions or other information contained in this message may not be that of the organization. ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/
Re: [c-nsp] Understanding 10G line card oversubscription
There is the 4 port group which I believe is the same as a port pair on a 6708. Ie. No free switching. Based on some comments by Tim Stevenson, the link between the four port group and the fabric asic is 10G rather than the 16G in the 6708. But he mentions some of the same chips (metro, r2d2) so the architecture can't be much different from the 6708. Then there is the fabric group which is switched locally without going over the fabric. show asic slot sh int capabilities | inc ASIC Of course a diagram would be really helpful but those usually come with an NDA. 6708 results: ASIC Name Count Version KUMA 2 (3.0) METRO_ARGOS 2 (3.0) METRO_KRYPTON 2 (3.0) SSA 2 (9.0) R2D2 8 (2.0) TIANGANG 4 (54.0) Te/1 Ports-in-ASIC (Sub-port ASIC) : 1,4-5,7 (1) Te/2 Ports-in-ASIC (Sub-port ASIC) : 2-3,6,8 (2) My understanding of the chips is as follows: R2D2(per port) -> Tiangang (port pair) -> Metro argos -> SSA (Fabric complex) Metro Krypto interfaces to the EARL complex aka PFC (local switching/routing to avoid fabric) Kuma acts as a bus bridge (not relevant to fabric switching) The assumption is that the Tiangang is replace with another chip that can handle 4 ports. I assume the new FPGA - Metro link is somehow different on the 6716 but I don't know how. The 6704 skips the tiangang chips and directly connects the R2D2 to the Metro. >From a Cisco doc >(http://www.cisco.com/en/US/docs/ios/vswitch/command/reference/vs_02.pdf): ASIC Name Count Version KUMA 2 (2.0) METRO_ARGOS 2 (2.0) METRO_KRYPTON 2 (2.0) SSA 2 (8.0) R2D2 4 (2.0) Short answer seems to be that the 6716 is basically the same despite claims to the contrary Mack -Original Message- From: cisco-nsp-boun...@puck.nether.net [mailto:cisco-nsp-boun...@puck.nether.net] On Behalf Of Phil Mayers Sent: Monday, March 21, 2011 1:51 PM To: cisco-nsp@puck.nether.net Subject: Re: [c-nsp] Understanding 10G line card oversubscription On 03/21/2011 06:33 PM, Mack McBride wrote: > The 6716 is going to have similar limitations but I don't have a good > document on how the port asics connect My understanding is that the 6716 is quite different from the 6708. There's no "free" local switching within port groups AFAIK. The differences have been discussed on the list before. > > If someone can verify the connection method and limitations on the 6716 it > would be appreciated. Do you have specific IOS commands? We've got a few in service. ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/ ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/
Re: [c-nsp] Understanding 10G line card oversubscription
On 03/21/2011 06:33 PM, Mack McBride wrote: The 6716 is going to have similar limitations but I don't have a good document on how the port asics connect My understanding is that the 6716 is quite different from the 6708. There's no "free" local switching within port groups AFAIK. The differences have been discussed on the list before. If someone can verify the connection method and limitations on the 6716 it would be appreciated. Do you have specific IOS commands? We've got a few in service. ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/
Re: [c-nsp] Understanding 10G line card oversubscription
The 6708 has an odd port layout. Pairs of ports connect up to an FPGA which has 16GB of bandwidth to the fabric asic. The port pairs are 1,4;5,7;2,3; and 6,8 These pairs of ports can only send or receive 16GB in total. The fabric asic has 20G and these combine two pairs: 1,4,5,7 and 2,4,6,8 Traffic between ports in these groups does not go over the fabric and is not counted against that BW. The danger of packet loss is extremely dependent on burstiness of the traffic. A general rule of thumb is over 80% of channel capacity is bad and will result in drops. But that will not hold if the traffic is buffered or rate-limited somewhere else. If the 20G fabric is congested 'really bad things' can happen. This is due to lost control traffic. This is usually indicated by extremely erratic traffic and potentially even blade resets. You will see input queue drops across the box if the fabric becomes congested on any blade. The maximum you probably want to push in a single direction is 12.8G on a port pair and 16G on a port group. Keep in mind communication between ports in a group is excluded in the group but included in the pair. The 6716 is going to have similar limitations but I don't have a good document on how the port asics connect to the fabric asic. On the 6716 the ports are more normally ordered 1,2,3,4;5,6,7,8;9,10,11,12;13,14,15,16. My guess (and it is only a guess) is that the quads of ports are connected down to the fabric asic via a single FPGA and probably have the same 16G limit as the port pairs. I can further speculate that ports 1-8 are on one fabric channel (fabric 1) and ports 9-16 are on the other fabric channel (fabric 0). If someone can verify the connection method and limitations on the 6716 it would be appreciated. Mack McBride Network Architect -Original Message- From: cisco-nsp-boun...@puck.nether.net [mailto:cisco-nsp-boun...@puck.nether.net] On Behalf Of D B Sent: Sunday, March 20, 2011 2:13 PM To: cisco-nsp@puck.nether.net Subject: [c-nsp] Understanding 10G line card oversubscription I'm looking for in-depth documentation on switch fabric oversubscription for these two 10G line cards: WS-X6708-10GE WS-X6716-10GE I'd like to understand its design and operation regarding these cards. Also, how to identify/quantify instances where oversubscription is nearing/hitting thresholds that will cause packet drops (SNMP?). ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/ ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/
[c-nsp] Understanding 10G line card oversubscription
I'm looking for in-depth documentation on switch fabric oversubscription for these two 10G line cards: WS-X6708-10GE WS-X6716-10GE I'd like to understand its design and operation regarding these cards. Also, how to identify/quantify instances where oversubscription is nearing/hitting thresholds that will cause packet drops (SNMP?). ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/
