[c-nsp] Equal Cost Routing w/ 3550
I'm using a 3550-12t with routed ports. Two of the ports are configured with the default route like this: ip route 0.0.0.0 0.0.0.0 Gig 0/1 ip route 0.0.0.0 0.0.0.0 Gig 0/2 'show ip route 0.0.0.0' indicates: Routing entry for 0.0.0.0/0, supernet Known via "static", distance 1, metric 0, candidate default path Routing Descriptor Blocks: * , via GigabitEthernet0/1 Route metric is 0, traffic share count is 1 , via GigabitEthernet0/2 Route metric is 0, traffic share count is 1 My first sign of trouble is that the '*' fails to switch back-n-forth between the two available routes. All traffic is directed to the interface with the '*' and it never changes. The second interface has zero outgoing traffic. Both links show incoming traffic from the remote routers. This was working before until I moved my equipment to another cabinet with new links (and new IP addresses and possibly different routers at the remote end). So ... what are my symptoms: a) no outgoing traffic on one interface b) users complain of sporadic connectivity issues, slow network access I have posted the output from some troubleshooting commands here: http://www.hectorh.com/cisco-output.txt Thank you for your help! -- Hector Herrera ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/
[c-nsp] Router recommendation for load balancing setup
Hello, I'm looking for a router that can: - handle load-balancing on two 100Mbps links with minimal cpu impact - must have at least 4 ports, at least 2 of which should be GigE and the other two must support FE or GigE - BGP with 25,000 routes My budget is small (under $2,000) so I'm probably looking for EOL/EOS products. I'm currently using a 3550-12t for the task, with the only drawback that the cpu hits 99% load with a 5000 packets per sec./40Mbps combined throughput on the load-balanced links. The two 100Mbps uplinks never reach more than 50% utilization because the router can't handle the load. I would like to be able to handle up to 80% utilization on the 100Mbps links. Thank you for your suggestions, Hector ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/
Re: [c-nsp] Bug ID CSCsv50653
On Wed, Jan 6, 2010 at 2:03 PM, Jeff Kell wrote: > On 1/6/2010 4:55 PM, Jason Shearer wrote: >> After reload, 3550 does not load share >> >> 1st Found-In >> 12.2(35)SE >> Known Affected Versions >> >> >> Fixed-In >> 12.2(50)SE >> 12.2(50)SE1 >> > > Well, that's a major crock-o-stuff, as 12.2(46)SE6 is the last > officially supported/provided IOS release for that platform (other than > the DC version). > > Jeff Yes, that is quite ugly. I'm currently using 12.2(50)SE3 on a 3550-12T and the only difficulties that I have run into is a high ( >90% cpu load when total throughput on the load-balanced links reaches 200 Mbps ). I am curious to find out if the high cpu load is caused by some incompatibility between 12.2(50)SE3 and the 3550-12T (since the version is not officially supported on the platform). However, this bug (no load sharing after reload) is making me think twice about testing 12.2(46)SE6. On the other hand, the bug fix for this issue could be the reason for the high cpu load Out of curiosity, is anybody here using a 3550 to route more than 200Mbps ( at about 40,000 packets per second forwarding rate ), I would be interested in comparing cpu loads with or without load-sharing. Thank you for all the copies of the bug that I received (both to the list and privately). -- Hector Herrera ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/
[c-nsp] Bug ID CSCsv50653
I don't have access to the bug toolkit. Could someone please send the details on this bug: CSCsv50653 I want to know if it is affecting my load-balancing setup. Thank you -- Hector ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/
Re: [c-nsp] 3550 High CPU - nothing in proc cpu
On Sun, Nov 22, 2009 at 4:01 PM, e ninja wrote: > Hector, > > It is interesting that the cisco article tells you how to profile your cpu > but not how to interpret the results ;-) > > There is only one way to interpret the results - contact Cisco to report the > abnormality. They will have to decode the address/es using the symbol files > for your device software which will reveal the culprit function/s. It should > be pretty straight forward to isolate cause and rectify thereafter. I did receive an email from someone at Cisco offering to look up the functions. Thank you :-) I can't wait to see the outcome. > FYI, seeing CPU spikes to X% during high traffic is not abnormal for most > non-distributed platforms that are groaning under an inappropriate switching > algorithm or overload. > > Out of curiosity, is 40% cpu utilization above your benchmarked baseline? If > no, ignore. Also, any alignment corrections? device#sh align Your question made me go back and review my notes. CPU load appears to be directly correlated to the amount of traffic on the switch. At 50Mbps the cpu load is 40%, at 200Mbps the load is 100%. At 20Mbps the load (currently) is 10% I wonder if expecting the 3550-12t platform to handle more than 200Mbps is too much to ask? The specs indicate it's capable of 17Mpps. According to the logs, at 200Mbps (with the 100% cpu load) the router was forwarding 45Kpps, much less than the advertised capacity. Perhaps it is a bad design on my part. I learned that the 3550-12t has three forwarding engines, one for each set of four interfaces (0/1 to 0/4, 0/5 to 0/8 and 0/9 to 0/12) With that in mind, I configured a VRF with four routed interfaces (0/1 to 0/4). 0/3 is a BGP interface. 0/4 is the LAN. 0/1 and 0/2 are configured in a load-balancing static default route. The forwarding engine is configured to use per-destination load-balancing. If I understand it correctly, Cisco's load-balancing in per-destination mode has an initial cost when the destination is not present in the routing table, but once it is there, CEF takes care of the forwarding. Since the traffic on the network is stream based (Live video streams), with very few new destinations (less than 500 per hour), but a constant stream of packets which should be handled by CEF. So I'm still at a loss ... Should I expect better performance from the 3550-12t or am I trying to squeeze blood out of stones? Hector ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/
Re: [c-nsp] 3550 High CPU - nothing in proc cpu
I had another opportunity to debug the high cpu usage on the 3550-12t. show proc cpu indicated that cpu load was 39% interrupt, 40% total So it's definitively a high interrupt rate that is using up the cpu. I also debugged the switching mechanism, and although I have high amounts of TTL-expired events, they only occur at a rate of 2-3 per second. I proceeded to profile the cpu usage with: profile profile start ... 10 mins later profile stop show profile terse Granularity was 8 due to the largest free block being about half the size of the main:text section. This gave me a listing of all the memory ranges and a count of how many times the cpu was found to be in that memory location. System Total = 000141506 Interrupt Total = 56163 (39 percent) Sched Total = 94547 (66 percent) Interrupt [00] = 56163 (39 percent) The interrupt breakdown is (top 3): 0x475F50 with 3281 counts (~5.4 per sec.) 0x4B82B8 with 1667 counts (~2.7 per sec) 0x4B8F90 with 1456 counts (~2.4 per sec) My question is: How do I convert those memory addresses into something that would tell me what interrupts are being triggered so much? Thank you, Hector ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/
Re: [c-nsp] 3550 High CPU - nothing in proc cpu
Great, so now I know: from 'show ip cef switching stat' I learned that there is a large number of packets with an expired TTL (TTL-expired is handled by the IP process, ie. software routing) from 'show interface switching' (hidden command) I learned the interface that has a high number of packets In and packets Out in the row "IP Process" Since the number of packets in the two commands above are very close to each other, I think I have identified the network interface with the large number of TTL-expired packets. It is a BGP interface, so my best guess is that a BGP neighbour is advertising routes that they don't actually carry in their routing tables and for some reason they are sending the packets back to me, and the question now is to locate the culprit route advertisement and contact the neighbor. Right? Still, for the next time I see high cpu usage, the commands to use are: 'show process cpu' and look at the first few lines to determine if it's interrupts or processes consuming the cpu time. If it's processes, look at the list of processes for any that are using large percentages. To diagnose high cpu consumption by interrupts, CPU Profiling (http://www.cisco.com/en/US/products/hw/routers/ps359/products_tech_note09186a00801c2af0.shtml) is a possible tool. Thank you all for your help! Hector ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/
Re: [c-nsp] 3550 High CPU - nothing in proc cpu
Thank you for your responses. I collected the commands to run the next time the cpu utilization spikes. I did manage to capture the output of 'show cef not-cef-switched' and it shows a very large number under the "unsupported" column. All the other columns are zero. Reading on the list archives I found a few commands to diagnose the "unsupported" column and according to the output, it appears that it's caused by TTL-expired being send to the cpu for processing. Does this mean that the hardware can't handle the TTL expired load or that TTL-expired messages are strictly a software process on this hardware (3550-12t)? If I have such a large number of TTL-expired messages, does that mean I have a routing loop somewhere? If so, I have three uplink interfaces, how do I find out which interface is causing the punts? Here is the output from the commands I ran: van-hc16-423-router#show ip cef switching stat Reason Drop Punt Punt2Host RP LES No route 0 0 37 RP LES Packet destined for us 0 273716 0 RP LES No adjacency8587 0 0 RP LES TTL expired0 01676276 RP LES Unclassified reason1 0 0 RP LES Neighbor resolution req 210055 3 0 RP LES Total 218643 2737191676313 AllTotal 218643 2737191676313 van-hc16-423-router#show ip cef switching stat feature IPv4 CEF input features: FeatureDropConsume Punt Punt2Host Gave route Total0 0 0 0 0 IPv4 CEF output features: FeatureDropConsume Punt Punt2HostNew i/f Total0 0 0 0 0 IPv4 CEF post-encap features: FeatureDropConsume Punt Punt2HostNew i/f Total0 0 0 0 0 IPv4 CEF for us features: FeatureDropConsume Punt Punt2HostNew i/f Total0 0 0 0 0 IPv4 CEF punt features: FeatureDropConsume Punt Punt2HostNew i/f Total0 0 0 0 0 IPv4 CEF local features: FeatureDropConsume Punt Punt2Host Gave route Total0 0 0 0 0 van-hc16-423-router#sh ip arp summ 16 IP ARP entries, with 0 of them incomplete van-hc16-423-router#sh sdm prefer The current template is the routing extended-match template. The selected template optimizes the resources in the switch to support this level of features for 16 routed interfaces and 1K VLANs. number of unicast mac addresses: 6K number of igmp groups: 6K number of qos aces:1K number of security aces: 1K number of unicast routes: 12K number of multicast routes:6K van-hc16-423-router#sh ip route summary IP routing table name is Default-IP-Routing-Table(0) IP routing table maximum-paths is 32 Route SourceNetworksSubnets OverheadMemory (bytes) connected 0 1 64 152 static 0 0 0 0 bgp 42800 0 0 0 External: 0 Internal: 0 Local: 0 internal1 1172 Total 1 1 64 1324 van-hc16-423-router#sh ip route vrf PublicRouter sum van-hc16-423-router#sh ip route vrf PublicRouter summary IP routing table name is PublicRouter(1) IP routing table maximum-paths is 32 Route SourceNetworksSubnets OverheadMemory (bytes) connected 0 4 256 608 static 1 0 128 152 bgp 428012741134154112 367036 External: 2408 Internal: 0 Local: 0 internal66 77352 Total 13411138154496 445148 van-hc16-423-router# On Sat, Nov 14, 2009 at 6:59 PM, Harald Firing Karlsen wrote: > Hector Herrera wrote: >> >> During a high network usage event, the cpu load increased to 90% >> sustained, while a 'show processes cpu' did not reveal any culprits. >> I suspected IP Input may be consuming a high amount of cpu, but it was >> only at 2.7% >> >> The 3550 is working as a L3 router with two static entries for the >> default gw (for load balancing on our uplink). >> >> Traffic levels at the time of the high cpu usage were ~120Mbps. >> >> I also
[c-nsp] 3550 High CPU - nothing in proc cpu
During a high network usage event, the cpu load increased to 90% sustained, while a 'show processes cpu' did not reveal any culprits. I suspected IP Input may be consuming a high amount of cpu, but it was only at 2.7% The 3550 is working as a L3 router with two static entries for the default gw (for load balancing on our uplink). Traffic levels at the time of the high cpu usage were ~120Mbps. I also examined broadcast packet counts and traffic destined for the router itself. They also did not reveal anything out of the ordinary. Do you have any suggestions on what I should be looking at to determine the source of the high cpu usage? Thank you, Hector ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/
Re: [c-nsp] Hidiing a traceroute
On Sat, Oct 10, 2009 at 12:21 PM, Jason Alex wrote: > Dear All, > I want to hide a traceroute hops inside my network > i know you can hide the traceroute inside an MPLS network > > can we hide also the traceroute inside an IP network > > Thanks In advance > > Regards > Jason > CCIE#24775 An MPLS network hides the network hops because as far as the packet is concerned, the MPLS network is a tunnel with no router hops. To hide a traceroute inside a L3 network, you need to block ICMP TTL-expired messages from the hops you want to hide. However, the hops will still be visible since every router decrements the TTL by one, and the traceroute source will notice it is missing TTL-expired messages from your hidden hops. Hector ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/
Re: [c-nsp] FTP seems to work
On Fri, Sep 25, 2009 at 9:36 AM, Jason Gurtz wrote: > I was about to write a little perl to further address the recent outcry > over the cisco.com Java misfeatures when lo, I discovered > ftp://download-sj.cisco.com will accept my cco login id/pass. I poked > around and discovered /cisco/ios and /cisco/ciscosecure/pix seemed to have > what I'd be looking for. > > Is this new or just a secret DL feature? > > ~JasonG Either it's now fixed, or my account without a maintenance contract can't see it. Hector ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/
Re: [c-nsp] Which IP's belong to AS1234?
On Thu, Sep 24, 2009 at 11:55 PM, Andy Saykao wrote: > This might be a silly question but is there a tool somewhere that will > give me a list of IP's that are owned by a particular AS. > > As an example, I might want to know which IP blocks belong to AS1234? > > Thanks. > > Andy Have you tried robtex? http://www.robtex.com/as/as1234.html#bgp Hector ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/
