Re: [c-nsp] 3845 maxing out at 400 Mbps
Just an update -- I shuffled interfaces last night, so that the bulk of the traffic goes through the 3845's native GigE interfaces. This has reduced CPU by about 10 to 15% and my new max is 423 Mbps, already 5% my previous max. Thanks for the help. Frank -Original Message- From: Pierre Emeriaud [mailto:petrus...@gmail.com] Sent: Tuesday, March 29, 2011 12:18 PM To: frnk...@iname.com Cc: cisco-nsp@puck.nether.net Subject: Re: [c-nsp] 3845 maxing out at 400 Mbps 2011/3/29 Frank Bulk : > Yes, I am running that HWIC! > > NAME: "High Speed WAN Interface Card - 1 Port Gigabit Ethernet on Slot 0 > SubSlot 2", DESCR: "High Speed WAN Interface Card - 1 Port Gigabit Ethernet" > PID: HWIC-1GE-SFP , VID: V01 , SN: > > If I shuffle around interfaces so that the inter-3845 links use the HWIC > instead, would that totally resolve the issue? I guess so. This issue was discussed some time ago on frnog mailing list (french nanog) After asking earlier today, I told myself that you would already use the hwic for the inter-3845, but it appears not. Things should improve if less traffic flows through the hwic. regards, -pierre. ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/
Re: [c-nsp] 3845 maxing out at 400 Mbps
2011/3/29 Frank Bulk : > Yes, I am running that HWIC! > > NAME: "High Speed WAN Interface Card - 1 Port Gigabit Ethernet on Slot 0 > SubSlot 2", DESCR: "High Speed WAN Interface Card - 1 Port Gigabit Ethernet" > PID: HWIC-1GE-SFP , VID: V01 , SN: > > If I shuffle around interfaces so that the inter-3845 links use the HWIC > instead, would that totally resolve the issue? I guess so. This issue was discussed some time ago on frnog mailing list (french nanog) After asking earlier today, I told myself that you would already use the hwic for the inter-3845, but it appears not. Things should improve if less traffic flows through the hwic. regards, -pierre. ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/
Re: [c-nsp] 3845 maxing out at 400 Mbps
Let me clear the air and say that I'm very happy with the performance of our 3845s -- they've done much better than even our consulting company thought they would. They need to last just a few more weeks until the new border routers we ordered come in and we turn them up. I just need to buy a few more weeks' time. What prompted the initial question was that I was seeing different behavior between the two (identical) routers. That gave me a unique opportunity to compare and contrast. I think the HWIC is likely the culprit here. Frank -Original Message- From: Christopher Pilkington [mailto:c...@0x1.net] Sent: Tuesday, March 29, 2011 11:55 AM To: Asbjorn Hojmark - Lists Cc: frnk...@iname.com; Cisco NSP Subject: Re: [c-nsp] 3845 maxing out at 400 Mbps On Tue, Mar 29, 2011 at 4:00 AM, Asbjorn Hojmark - Lists wrote: > Actually, the 3800 is positioned for T3/E3 speeds... I consider it > quite impressive that you're pushing up to 400 Mbps though them with > some features. I believe the marketing blurb for the 3845 was "full T3 with concurrent services." Even this is a stretch of reality. A voice-heavy traffic mix (avg 300 bytes/packet) while using GRE/IPSec brings the 3845, and it's younger sibling the 3945, to it's knees around 35-40Mb/s. ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/
Re: [c-nsp] 3845 maxing out at 400 Mbps
On Tue, Mar 29, 2011 at 4:00 AM, Asbjorn Hojmark - Lists wrote: > Actually, the 3800 is positioned for T3/E3 speeds... I consider it > quite impressive that you're pushing up to 400 Mbps though them with > some features. I believe the marketing blurb for the 3845 was "full T3 with concurrent services." Even this is a stretch of reality. A voice-heavy traffic mix (avg 300 bytes/packet) while using GRE/IPSec brings the 3845, and it's younger sibling the 3945, to it's knees around 35-40Mb/s. ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/
Re: [c-nsp] 3845 maxing out at 400 Mbps
Yes, I am running that HWIC! NAME: "High Speed WAN Interface Card - 1 Port Gigabit Ethernet on Slot 0 SubSlot 2", DESCR: "High Speed WAN Interface Card - 1 Port Gigabit Ethernet" PID: HWIC-1GE-SFP , VID: V01 , SN: If I shuffle around interfaces so that the inter-3845 links use the HWIC instead, would that totally resolve the issue? Frank -Original Message- From: Pierre Emeriaud [mailto:petrus...@gmail.com] Sent: Tuesday, March 29, 2011 4:27 AM To: frnk...@iname.com Cc: cisco-nsp@puck.nether.net Subject: Re: [c-nsp] 3845 maxing out at 400 Mbps Hi Frank, all, 2011/3/29 Frank Bulk : > We have two 3845's as border routers, each with three GigE interfaces (one > facing upstream, the other downstream, the third facing the other 3845). > The first 3845 has a typical packet-size mix (residential/business Internet) > is consistently maxing out at 400 Mbps (predominately ingress because of > asymmetric routing) running at about 43 kpps and 40% CPU. It's flat-lines > very evenly, uncannily so. We checked and double-checked transport and it's > set much higher, the same as the second 3845. I guess you're using a hwic-1ge-sfp in your 3845 ? Quote from http://www.cisco.com/en/US/prod/collateral/routers/ps5854/prod_qas0900aecd80 169bf0_ps5855_Products_Q_and_A_Item.html : Q. What is the maximum throughput on the Gigabit Ethernet HWIC? A. The HWIC bus interface is limited to 400 Mbps of full duplex. The actual throughput of the Gigabit Ethernet HWIC is limited by the throughput of individual platforms. Under bidirectional traffic of 1518 bytes or larger, the Gigabit Ethernet HWIC can support up to an aggregate of 350 Mbps on Cisco 2811 and 2821 routers, 400 Mbps on Cisco 2851 routers, and 500 Mbps on Cisco 3800 Series platforms. So your 400Mbps limit could be hardware and not from configuration or software. Regards, Pierre. ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/
Re: [c-nsp] 3845 maxing out at 400 Mbps
I agree. It's just that I have an identical router that's set up identically with slightly lower ingress but higher total ingress + egress numbers, and can go over 40%. Frank -Original Message- From: Asbjorn Hojmark - Lists [mailto:li...@hojmark.org] Sent: Tuesday, March 29, 2011 3:01 AM To: frnk...@iname.com Cc: Cisco NSP Subject: Re: [c-nsp] 3845 maxing out at 400 Mbps On Mon, 28 Mar 2011 21:14:21 -0500, you wrote: > The ACLs are BCP 38-oriented with eBGP; no rate-limiting. We're running > 124-11.XW2. You really should look at upgrading that to some more recent and less End-of-X. 12.4 XW also has know vulnerabilities only fixed in later releases. > Any ideas? The numbers are well below Cisco's router spec sheet. Actually, the 3800 is positioned for T3/E3 speeds... I consider it quite impressive that you're pushing up to 400 Mbps though them with some features. The spec sheet is best case numbers with no features. *Any* feature that you turn on will negatively affect performance, and the actual performance hit for each feature will also vary with traffic patterns. -A ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/
Re: [c-nsp] 3845 maxing out at 400 Mbps
The 'sh ip cef switching stats' is unfortunately not supported on my IOS release. AFAIK, this is not BU-special software. I d/l this several years ago from using CCO account when initially turning these up. I'd prefer not upgrade unless someone can point me to a specific issue/bug fix. Thanks for your thoughts. Frank -Original Message- From: cisco-nsp-boun...@puck.nether.net [mailto:cisco-nsp-boun...@puck.nether.net] On Behalf Of Lukasz Bromirski Sent: Tuesday, March 29, 2011 2:33 AM To: cisco-nsp@puck.nether.net Subject: Re: [c-nsp] 3845 maxing out at 400 Mbps On 2011-03-29 04:14, Frank Bulk wrote: > We have two 3845's as border routers, each with three GigE interfaces (one > facing upstream, the other downstream, the third facing the other 3845). > The first 3845 has a typical packet-size mix (residential/business Internet) > is consistently maxing out at 400 Mbps (predominately ingress because of > asymmetric routing) running at about 43 kpps and 40% CPU. It's flat-lines > very evenly, uncannily so. We checked and double-checked transport and it's > set much higher, the same as the second 3845. > The second 3845, which has a mix of both ingress and egress traffic at a > combined 82 kpps (35 kpps ingress/50 kpps egress) but lower combined 360 > Mbps operates at a higher CPU (presumably because there's also egress > traffic) with no flatlining. Are there any CEF drops? Have you checked 'sh ip cef switching stats'? > The ACLs are BCP 38-oriented with eBGP; no rate-limiting. > We're running 124-11.XW2. Why you're running BU-special software? Some specific feature not included in normal IOS? Given the relese date of the software, all the features should be already in the mainline IOS. You should propably move to 12.4(15)T or 15.0(1)Mx (latest rebuild). -- "There's no sense in being precise when | Łukasz Bromirski you don't know what you're talking | jid:lbromir...@jabber.org about." John von Neumann |http://lukasz.bromirski.net ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/ ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/
Re: [c-nsp] 3845 maxing out at 400 Mbps
On 3/28/2011 11:05 PM, Frank Bulk wrote: Packet sizes are believed to be roughly equivalent between both 3845's because our upstream is just preffing some subnets toward one path than another. I checked everything CEF/interface related on both routers and it all appears to be correct and healthy. Thanks, Frank Well they definitely are not. You have roughly the same throughput and almost double the packet rate on the second box. Also as someone else said I'm guessing you are using the gig card and that's the one that bounces off of 400mbps? The 3825/45s come with 2 built in gig ports only. What's your expectation on throughput on these boxes? Quite frankly, I think you are doing very well. tv ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/
Re: [c-nsp] 3845 maxing out at 400 Mbps
Hi Frank, all, 2011/3/29 Frank Bulk : > We have two 3845's as border routers, each with three GigE interfaces (one > facing upstream, the other downstream, the third facing the other 3845). > The first 3845 has a typical packet-size mix (residential/business Internet) > is consistently maxing out at 400 Mbps (predominately ingress because of > asymmetric routing) running at about 43 kpps and 40% CPU. It's flat-lines > very evenly, uncannily so. We checked and double-checked transport and it's > set much higher, the same as the second 3845. I guess you're using a hwic-1ge-sfp in your 3845 ? Quote from http://www.cisco.com/en/US/prod/collateral/routers/ps5854/prod_qas0900aecd80169bf0_ps5855_Products_Q_and_A_Item.html : Q. What is the maximum throughput on the Gigabit Ethernet HWIC? A. The HWIC bus interface is limited to 400 Mbps of full duplex. The actual throughput of the Gigabit Ethernet HWIC is limited by the throughput of individual platforms. Under bidirectional traffic of 1518 bytes or larger, the Gigabit Ethernet HWIC can support up to an aggregate of 350 Mbps on Cisco 2811 and 2821 routers, 400 Mbps on Cisco 2851 routers, and 500 Mbps on Cisco 3800 Series platforms. So your 400Mbps limit could be hardware and not from configuration or software. Regards, Pierre. ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/
Re: [c-nsp] 3845 maxing out at 400 Mbps
On 2011-03-29 04:14, Frank Bulk wrote: We have two 3845's as border routers, each with three GigE interfaces (one facing upstream, the other downstream, the third facing the other 3845). The first 3845 has a typical packet-size mix (residential/business Internet) is consistently maxing out at 400 Mbps (predominately ingress because of asymmetric routing) running at about 43 kpps and 40% CPU. It's flat-lines very evenly, uncannily so. We checked and double-checked transport and it's set much higher, the same as the second 3845. The second 3845, which has a mix of both ingress and egress traffic at a combined 82 kpps (35 kpps ingress/50 kpps egress) but lower combined 360 Mbps operates at a higher CPU (presumably because there's also egress traffic) with no flatlining. Are there any CEF drops? Have you checked 'sh ip cef switching stats'? The ACLs are BCP 38-oriented with eBGP; no rate-limiting. > We're running 124-11.XW2. Why you're running BU-special software? Some specific feature not included in normal IOS? Given the relese date of the software, all the features should be already in the mainline IOS. You should propably move to 12.4(15)T or 15.0(1)Mx (latest rebuild). -- "There's no sense in being precise when | Łukasz Bromirski you don't know what you're talking | jid:lbromir...@jabber.org about." John von Neumann |http://lukasz.bromirski.net ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/
Re: [c-nsp] 3845 maxing out at 400 Mbps
On Mon, 28 Mar 2011 21:14:21 -0500, you wrote: > The ACLs are BCP 38-oriented with eBGP; no rate-limiting. We're running > 124-11.XW2. You really should look at upgrading that to some more recent and less End-of-X. 12.4 XW also has know vulnerabilities only fixed in later releases. > Any ideas? The numbers are well below Cisco's router spec sheet. Actually, the 3800 is positioned for T3/E3 speeds... I consider it quite impressive that you're pushing up to 400 Mbps though them with some features. The spec sheet is best case numbers with no features. *Any* feature that you turn on will negatively affect performance, and the actual performance hit for each feature will also vary with traffic patterns. -A ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/
Re: [c-nsp] 3845 maxing out at 400 Mbps
Packet sizes are believed to be roughly equivalent between both 3845's because our upstream is just preffing some subnets toward one path than another. I checked everything CEF/interface related on both routers and it all appears to be correct and healthy. Thanks, Frank -Original Message- From: cisco-nsp-boun...@puck.nether.net [mailto:cisco-nsp-boun...@puck.nether.net] On Behalf Of Tony Varriale Sent: Monday, March 28, 2011 10:24 PM To: cisco-nsp Subject: Re: [c-nsp] 3845 maxing out at 400 Mbps On 3/28/2011 9:14 PM, Frank Bulk wrote: > We have two 3845's as border routers, each with three GigE interfaces (one > facing upstream, the other downstream, the third facing the other 3845). > The first 3845 has a typical packet-size mix (residential/business Internet) > is consistently maxing out at 400 Mbps (predominately ingress because of > asymmetric routing) running at about 43 kpps and 40% CPU. It's flat-lines > very evenly, uncannily so. We checked and double-checked transport and it's > set much higher, the same as the second 3845. > > The second 3845, which has a mix of both ingress and egress traffic at a > combined 82 kpps (35 kpps ingress/50 kpps egress) but lower combined 360 > Mbps operates at a higher CPU (presumably because there's also egress > traffic) with no flatlining. > > The ACLs are BCP 38-oriented with eBGP; no rate-limiting. We're running > 124-11.XW2. > > Any ideas? The numbers are well below Cisco's router spec sheet. > > Frank The first idea is pretty obvious: different packet sizes. Why so? The second idea would be to make sure you are staying in the CEF path as much as possible. Verify that yet? tv ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/ ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/
Re: [c-nsp] 3845 maxing out at 400 Mbps
On 3/28/2011 9:14 PM, Frank Bulk wrote: We have two 3845's as border routers, each with three GigE interfaces (one facing upstream, the other downstream, the third facing the other 3845). The first 3845 has a typical packet-size mix (residential/business Internet) is consistently maxing out at 400 Mbps (predominately ingress because of asymmetric routing) running at about 43 kpps and 40% CPU. It's flat-lines very evenly, uncannily so. We checked and double-checked transport and it's set much higher, the same as the second 3845. The second 3845, which has a mix of both ingress and egress traffic at a combined 82 kpps (35 kpps ingress/50 kpps egress) but lower combined 360 Mbps operates at a higher CPU (presumably because there's also egress traffic) with no flatlining. The ACLs are BCP 38-oriented with eBGP; no rate-limiting. We're running 124-11.XW2. Any ideas? The numbers are well below Cisco's router spec sheet. Frank The first idea is pretty obvious: different packet sizes. Why so? The second idea would be to make sure you are staying in the CEF path as much as possible. Verify that yet? tv ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/
[c-nsp] 3845 maxing out at 400 Mbps
We have two 3845's as border routers, each with three GigE interfaces (one facing upstream, the other downstream, the third facing the other 3845). The first 3845 has a typical packet-size mix (residential/business Internet) is consistently maxing out at 400 Mbps (predominately ingress because of asymmetric routing) running at about 43 kpps and 40% CPU. It's flat-lines very evenly, uncannily so. We checked and double-checked transport and it's set much higher, the same as the second 3845. The second 3845, which has a mix of both ingress and egress traffic at a combined 82 kpps (35 kpps ingress/50 kpps egress) but lower combined 360 Mbps operates at a higher CPU (presumably because there's also egress traffic) with no flatlining. The ACLs are BCP 38-oriented with eBGP; no rate-limiting. We're running 124-11.XW2. Any ideas? The numbers are well below Cisco's router spec sheet. Frank ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/
