RE: Router / Protocol Problem
Good morning everyone. I just wanted to say thanks for all the help. I did discover the problem this morning and I should be hit with a herring. I upgraded the IOS on the router with the issue to match the other router and the problem was still there. So I tested and noticed the following line in the logs, since I was on console it popped up right in front of me. Sep 7 06:50:20.697 EST: %SEC-6-IPACCESSLOGP: list 166 denied tcp 69.50.222.8(25) - 69.4.74.14(2421), 4 packets What is this I thought? What is my ACL 166 doing this? I thought I tested removing all access-lists from interfaces with the original problem came up. Apparently not. Here is my ACL 166, the first line is what was being matched. Apparently some how this connection is being matched via NBAR for good old Code Red. access-list 166 deny ip any any dscp 1 log access-list 166 deny tcp any any eq sunrpc access-list 166 deny tcp any any eq 135 access-list 166 deny tcp any any eq 137 access-list 166 deny tcp any any eq 138 access-list 166 deny tcp any any eq 139 access-list 166 deny tcp any any eq 445 access-list 166 deny tcp any any eq 5554 access-list 166 deny tcp any any eq 9996 access-list 166 deny tcp any any eq 1025 access-list 166 deny udp any any eq 1434 access-list 166 deny udp any any eq 135 access-list 166 deny udp any any eq netbios-ns access-list 166 deny udp any any eq netbios-dgm access-list 166 deny udp any any eq netbios-ss access-list 166 deny udp any any eq 445 access-list 166 deny icmp any any redirect access-list 166 deny ip 127.0.0.0 0.255.255.255 any access-list 166 deny ip 10.0.0.0 0.255.255.255 any access-list 166 deny ip 172.16.0.0 0.15.255.255 any access-list 166 deny ip 192.168.0.0 0.0.255.255 any access-list 166 permit ip any any class-map match-any http-hacks match protocol http url *default.ida* match protocol http url *cmd.exe* match protocol http url *root.exe* policy-map mark-inbound-http-hacks class http-hacks set ip dscp 1 I have always had this on my FE0/0 as an outbound ACL, well atleast since Code Red came about: ip access-group 166 out. Now I have two questions. Is that not a good idea to have this on FE0/0 out? Second, why the heck would a smtp connection be matched via my http-hacks class-map? Thanks again everyone, Mike -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Rodney Dunn Sent: Wednesday, September 06, 2006 8:45 PM To: Christopher L. Morrow Cc: Rodney Dunn; Mike Walter; Hank Nussbacher; Justin M. Streiner; nanog@merit.edu Subject: Re: Router / Protocol Problem Then that proves it's not a local router problem then. :) On Wed, Sep 06, 2006 at 07:49:26PM +, Christopher L. Morrow wrote: On Wed, 6 Sep 2006, Rodney Dunn wrote: Get a sniffer trace. Packets on the wire prove what's going on. provided the packets get back to him, it seems his problem is traffic getting back to him :( so probably no packets will be on the wire (none in question atleast)...
RE: Router / Protocol Problem
At 07:27 AM 07-09-06 -0400, Mike Walter wrote: Best moved to cisco-nsp. -Hank Nussbacher http://www.interall.co.il Good morning everyone. I just wanted to say thanks for all the help. I did discover the problem this morning and I should be hit with a herring. I upgraded the IOS on the router with the issue to match the other router and the problem was still there. So I tested and noticed the following line in the logs, since I was on console it popped up right in front of me. Sep 7 06:50:20.697 EST: %SEC-6-IPACCESSLOGP: list 166 denied tcp 69.50.222.8(25) - 69.4.74.14(2421), 4 packets What is this I thought? What is my ACL 166 doing this? I thought I tested removing all access-lists from interfaces with the original problem came up. Apparently not. Here is my ACL 166, the first line is what was being matched. Apparently some how this connection is being matched via NBAR for good old Code Red. access-list 166 deny ip any any dscp 1 log access-list 166 deny tcp any any eq sunrpc access-list 166 deny tcp any any eq 135 access-list 166 deny tcp any any eq 137 access-list 166 deny tcp any any eq 138 access-list 166 deny tcp any any eq 139 access-list 166 deny tcp any any eq 445 access-list 166 deny tcp any any eq 5554 access-list 166 deny tcp any any eq 9996 access-list 166 deny tcp any any eq 1025 access-list 166 deny udp any any eq 1434 access-list 166 deny udp any any eq 135 access-list 166 deny udp any any eq netbios-ns access-list 166 deny udp any any eq netbios-dgm access-list 166 deny udp any any eq netbios-ss access-list 166 deny udp any any eq 445 access-list 166 deny icmp any any redirect access-list 166 deny ip 127.0.0.0 0.255.255.255 any access-list 166 deny ip 10.0.0.0 0.255.255.255 any access-list 166 deny ip 172.16.0.0 0.15.255.255 any access-list 166 deny ip 192.168.0.0 0.0.255.255 any access-list 166 permit ip any any class-map match-any http-hacks match protocol http url *default.ida* match protocol http url *cmd.exe* match protocol http url *root.exe* policy-map mark-inbound-http-hacks class http-hacks set ip dscp 1 I have always had this on my FE0/0 as an outbound ACL, well atleast since Code Red came about: ip access-group 166 out. Now I have two questions. Is that not a good idea to have this on FE0/0 out? Second, why the heck would a smtp connection be matched via my http-hacks class-map? Thanks again everyone, Mike -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Rodney Dunn Sent: Wednesday, September 06, 2006 8:45 PM To: Christopher L. Morrow Cc: Rodney Dunn; Mike Walter; Hank Nussbacher; Justin M. Streiner; nanog@merit.edu Subject: Re: Router / Protocol Problem Then that proves it's not a local router problem then. :) On Wed, Sep 06, 2006 at 07:49:26PM +, Christopher L. Morrow wrote: On Wed, 6 Sep 2006, Rodney Dunn wrote: Get a sniffer trace. Packets on the wire prove what's going on. provided the packets get back to him, it seems his problem is traffic getting back to him :( so probably no packets will be on the wire (none in question atleast)... +++ This Mail Was Scanned By Mail-seCure System at the Tel-Aviv University CC.
RE: Router / Protocol Problem
Apparently some how this connection is being matched via NBAR for good old Code Red. Best moved to cisco-nsp. What!? Network operator discovers that measures taken to mitigate an old network security measure, long past their sell-by date, are now causing random grief. Seems to me like bang on topic for NANOG. What other such temporary mitigating measures are still in place long after the danger has passed. Note, that Code RED was a both an application vulnerability and a network DDoS. Even though there are likely still many hosts running the vulnerable application, the number is not sufficient to cause another massive DDoD and measures taken to protect against this particular peculiar DDoS, really don't have a good technical reason to remain in place. This is probably also another instance of the well-known ops problem: We know how to get stuff deployed but we can't undeploy stuff because we are too busy deploying other stuff. --Michael Dillon
Re: Router / Protocol Problem
On Thu, 7 Sep 2006 07:27:16 -0400 Mike Walter [EMAIL PROTECTED] wrote: Sep 7 06:50:20.697 EST: %SEC-6-IPACCESSLOGP: list 166 denied tcp 69.50.222.8(25) - 69.4.74.14(2421), 4 packets [...] I'm not very familiar with NBAR or how to use it for CodeRed, but this first rule: access-list 166 deny ip any any dscp 1 log Seems dubious. So I'm not not sure what sets the codepoint to 01 by default, but apparently CodeRed does? Nevertheless, this seems like a very weak basis for determining whether something is malicious. access-list 166 deny tcp any any eq 5554 access-list 166 deny tcp any any eq 9996 access-list 166 deny tcp any any eq 1025 access-list 166 deny udp any any eq 1434 You may realize this, but I bet some of the rules above I bet are matching on the occasional legitimate packets. Particular the last four rules above. In fact, I bet the rule that matches on TCP destination port 1025 probably has a lot of falsepositives. I'm not sure what you're trying to do with some of them, but if it is to stop some sort of worm, presumably you know that it will also stop applications that happen to choose those source ports. Windows hosts and apps will probably match the 1025 rule fairly frequently, UDP and NTP will match the UDP rule occasionally and various things will the others more or less frequently depending on what traverses your net. Now I have two questions. Is that not a good idea to have this on FE0/0 out? Second, why the heck would a smtp connection be matched via my http-hacks class-map? You don't show the interface config, but my guess is that the SMTP- looking packet may have originally had a codepoint of 1 and didn't really have anything to do with your policy-map. John
Re: Router / Protocol Problem
[EMAIL PROTECTED] writes: Network operator discovers that measures taken to mitigate an old network security measure, long past their sell-by date, are now causing random grief. Seems to me like bang on topic for NANOG. Agreed. Rare that people do haircuts on router configs; they're tedious and can not be delegated to an intern or someone else who doesn't have historical context. I just cut a config by half by removing unused ACLs, and even that is fairly painful. What other such temporary mitigating measures are still in place long after the danger has passed. (?) It's been almost nine and a half years and was a short-lived problem, but I'll betcha that an announcement from AS 7007 will have reachability problems to a measurable fraction of the Internet. That would make a kind of cool experiment. Vinny, you listening? ---Rob
Re: Router / Protocol Problem
Hi John, John Kristoff wrote: On Thu, 7 Sep 2006 07:27:16 -0400 Mike Walter [EMAIL PROTECTED] wrote: Sep 7 06:50:20.697 EST: %SEC-6-IPACCESSLOGP: list 166 denied tcp 69.50.222.8(25) - 69.4.74.14(2421), 4 packets [...] I'm not very familiar with NBAR or how to use it for CodeRed, but this first rule: access-list 166 deny ip any any dscp 1 log Seems dubious. So I'm not not sure what sets the codepoint to 01 by default, but apparently CodeRed does? Nevertheless, this seems like a very weak basis for determining whether something is malicious. It's his NBAR config lower down that sets the dscp value: class-map match-any http-hacks match protocol http url *default.ida* match protocol http url *cmd.exe* match protocol http url *root.exe* policy-map mark-inbound-http-hacks class http-hacks set ip dscp 1 So, there's probably two things that could happen here: One, NBAR is incorrectly identifying the SMTP traffic as code red, or two, the SMTP traffic is already marked with dscp 1. If you've using these values internally in your own network then they should be reset on all externally received traffic. Sam
Re: Router / Protocol Problem
Yeah. Don't want any operational stuff here. Need to get back to who's got a free 300-baud dialup in Antwerp. Hank Nussbacher wrote: At 07:27 AM 07-09-06 -0400, Mike Walter wrote: Best moved to cisco-nsp. -Hank Nussbacher http://www.interall.co.il Good morning everyone. I just wanted to say thanks for all the help. I did discover the problem this morning and I should be hit with a herring. I upgraded the IOS on the router with the issue to match the other router and the problem was still there. So I tested and noticed the following line in the logs, since I was on console it popped up right in front of me. Sep 7 06:50:20.697 EST: %SEC-6-IPACCESSLOGP: list 166 denied tcp 69.50.222.8(25) - 69.4.74.14(2421), 4 packets What is this I thought? What is my ACL 166 doing this? I thought I tested removing all access-lists from interfaces with the original problem came up. Apparently not. Here is my ACL 166, the first line is what was being matched. Apparently some how this connection is being matched via NBAR for good old Code Red. access-list 166 deny ip any any dscp 1 log access-list 166 deny tcp any any eq sunrpc access-list 166 deny tcp any any eq 135 access-list 166 deny tcp any any eq 137 access-list 166 deny tcp any any eq 138 access-list 166 deny tcp any any eq 139 access-list 166 deny tcp any any eq 445 access-list 166 deny tcp any any eq 5554 access-list 166 deny tcp any any eq 9996 access-list 166 deny tcp any any eq 1025 access-list 166 deny udp any any eq 1434 access-list 166 deny udp any any eq 135 access-list 166 deny udp any any eq netbios-ns access-list 166 deny udp any any eq netbios-dgm access-list 166 deny udp any any eq netbios-ss access-list 166 deny udp any any eq 445 access-list 166 deny icmp any any redirect access-list 166 deny ip 127.0.0.0 0.255.255.255 any access-list 166 deny ip 10.0.0.0 0.255.255.255 any access-list 166 deny ip 172.16.0.0 0.15.255.255 any access-list 166 deny ip 192.168.0.0 0.0.255.255 any access-list 166 permit ip any any class-map match-any http-hacks match protocol http url *default.ida* match protocol http url *cmd.exe* match protocol http url *root.exe* policy-map mark-inbound-http-hacks class http-hacks set ip dscp 1 I have always had this on my FE0/0 as an outbound ACL, well atleast since Code Red came about: ip access-group 166 out. Now I have two questions. Is that not a good idea to have this on FE0/0 out? Second, why the heck would a smtp connection be matched via my http-hacks class-map? Thanks again everyone, Mike -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Rodney Dunn Sent: Wednesday, September 06, 2006 8:45 PM To: Christopher L. Morrow Cc: Rodney Dunn; Mike Walter; Hank Nussbacher; Justin M. Streiner; nanog@merit.edu Subject: Re: Router / Protocol Problem Then that proves it's not a local router problem then. :) On Wed, Sep 06, 2006 at 07:49:26PM +, Christopher L. Morrow wrote: On Wed, 6 Sep 2006, Rodney Dunn wrote: Get a sniffer trace. Packets on the wire prove what's going on. provided the packets get back to him, it seems his problem is traffic getting back to him :( so probably no packets will be on the wire (none in question atleast)... +++ This Mail Was Scanned By Mail-seCure System at the Tel-Aviv University CC. -- Requiescas in pace o email Ex turpi causa non oritur actio http://members.cox.net/larrysheldon/
RE: Router / Protocol Problem
On Sep 6, 2006, at 9:04 AM, Mike Walter wrote: Recently with no changes to my network, I have been having problems connecting to certain websites and mail servers. I am always able to ping the sites and trace route without error. If I telnet to port 80 or port 25 it does not connect. If I login to my router and telnet sourcing my each of Internet Providers ports, I am able to get to the sites. I have talked with all the providers and none Check a packet dump and see if your affected boxes are sending SWE (SYN with ECN enabled) instead of plain SYN packets. Some firewalls (at least default m0n0wall and older PIX) reject and dump ECN syn packets while allowing pure syns through. If your affected peer has you running through some weird filter that's dropping SWE packets, this would cause symptoms exactly as you're seeing - ping is fine, traceroute is fine, but TCP sessions never complete the handshake (as the receiving side never got the first SYN). - J
Re: Router / Protocol Problem
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Seems dubious. So I'm not not sure what sets the codepoint to 01 by default, but apparently CodeRed does? Nevertheless, this seems like a very weak basis for determining whether something is malicious. There is an elegant solution; administrators should set the evil bit on any malicious packets seeking egress; http://www.faqs.org/rfcs/rfc3514.html Quoting: 0x0 If the bit is set to 0, the packet has no evil intent. Hosts, network elements, etc., SHOULD assume that the packet is harmless, and SHOULD NOT take any defensive measures. (We note that this part of the spec is already implemented by many common desktop operating systems.) 0x1 If the bit is set to 1, the packet has evil intent. Secure systems SHOULD try to defend themselves against such packets. Insecure systems MAY chose to crash, be penetrated, etc. And now for something completely different... - -- The whole point of the Internet is that different kinds of computers can interoperate. Every time you see a web site that only supports certain browsers or operating systems, they clearly don't get it. -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.2.2 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFFAI/WPlSPhv5tocwRAnhrAJ40WgDRn+9fSPXa5U4qZGRRGRbjowCfbBxI AaDLCfYgGF1MjcieyDvuuME= =pibC -END PGP SIGNATURE-
Router / Protocol Problem
Title: Router / Protocol Problem I normally would not post to the group, but I am 100% stumped and have talked with peers with no luck. I have (2) Cisco 7204 Routers running BGP with 3 peers and HSRP. I am not doing anything special with BGP, pretty much a default config that has not changed in years. Recently with no changes to my network, I have been having problems connecting to certain websites and mail servers. I am always able to ping the sites and trace route without error. If I telnet to port 80 or port 25 it does not connect. If I login to my router and telnet sourcing my each of Internet Providers ports, I am able to get to the sites. I have talked with all the providers and none can find a problem. If I shut down one specific peer, everything works fine. So I keep thinking it was that peers problem some how. I have tested with just that peer up and I still can not connect. However, when talking with that peer, they are able to telnet from their network to the sites I can not reach. I don't know what else to check besides shutting down that peer. Which since it is under a 3 year contract, not an option. That isn't the real solution anyhow. Can anyone shed some light on or off-list? Thanks, Mike Walter
Re: Router / Protocol Problem
On 06/09/06, Mike Walter [EMAIL PROTECTED] wrote: I normally would not post to the group, but I am 100% stumped and have talked with peers with no luck. I have (2) Cisco 7204 Routers running BGP with 3 peers and HSRP. I am not doing anything special with BGP, pretty much a default config that has not changed in years. Recently with no changes to my network, I have been having problems connecting to certain websites and mail servers. I am always able to ping the sites and trace route without error. If I telnet to port 80 or port 25 it does not connect. If I login to my router and telnet sourcing my each of Internet Providers ports, I am able to get to the sites. I have talked with all the providers and none can find a problem. If I shut down one specific peer, everything works fine. So I keep thinking it was that peers problem some how. I have tested with just that peer up and I still can not connect. However, when talking with that peer, they are able to telnet from their network to the sites I can not reach. I don't know what else to check besides shutting down that peer. Which since it is under a 3 year contract, not an option. That isn't the real solution anyhow. Can anyone shed some light on or off-list? Give your peer a /32 to install on their access router, verify that return path is via them and have them do connectivity tests to your problem sites. If that checks out you step by step through it. Ask to be moved to a different access router, next change your hardware. /Tony -- Tony Sarendal - [EMAIL PROTECTED]IP/Unix -= The scorpion replied, I couldn't help it, it's my nature =-
Re: Router / Protocol Problem
On Wed, 6 Sep 2006, Mike Walter wrote: I normally would not post to the group, but I am 100% stumped and have talked with peers with no luck. I have (2) Cisco 7204 Routers running BGP with 3 peers and HSRP. I am not doing anything special with BGP, pretty much a default config that has not changed in years. Please provide details on both your default config and the hardware you're using. You say you have two Cisco 7204s - are these straight '04s, or 7204VXRs? What NPE(s) are you using, and how much memory is on them? The BGP you're getting from your peers - are you getting full routes from any of them? Do you have CEF enabled on these routers? What IOS version(s) are running on these routers? What else are they doing besides slinging BGP routes? Does the problem go away for a while if you reboot one router or the other? Without knowing any of this, it sounds like you might have NPE-225, -300, or -400 with 256 MB of RAM and you are running into memory exhaustion issues from carrying full routes. That's been a pretty popular topic on this list and others like cisco-nsp in the last 12 months :) At a minimum, what do the output of show mem summary and show ip bgp sum from each router show you? Have you seen other performance problems lately, such as things getting mysteriously slower, beyond the rachability issues you mentioned above? If so, check if CEF is still running (if it was configured in the first place). When a 7200 gets dangerously low on free memory and CEF is running, it may cannibalize the IP CEF process to try to conserve memory. Earlier 12.0 releases did this - I don't know if newer ones still do it. jms
RE: Router / Protocol Problem
One more thing, I can successfully do a tcptraceroute if that matters. Mike Walter From: tony sarendal [mailto:[EMAIL PROTECTED] Sent: Wednesday, September 06, 2006 9:32 AMTo: Mike WalterCc: nanog@merit.eduSubject: Re: Router / Protocol Problem On 06/09/06, Mike Walter [EMAIL PROTECTED] wrote: I normally would not post to the group, but I am 100% stumped and have talked with peers with no luck. I have (2) Cisco 7204 Routers running BGP with 3 peers and HSRP. I am not doing anything special with BGP, pretty much a default config that has not changed in years. Recently with no changes to my network, I have been having problems connecting to certain websites and mail servers. I am always able to ping the sites and trace route without error. If I telnet to port 80 or port 25 it does not connect. If I login to my router and telnet sourcing my each of Internet Providers ports, I am able to get to the sites. I have talked with all the providers and none can find a problem. If I shut down one specific peer, everything works fine. So I keep thinking it was that peers problem some how. I have tested with just that peer up and I still can not connect. However, when talking with that peer, they are able to telnet from their network to the sites I can not reach. I don't know what else to check besides shutting down that peer. Which since it is under a 3 year contract, not an option. That isn't the real solution anyhow. Can anyone shed some light on or off-list? Give your peer a /32 to install on their access router, verify that return pathis via them and have them do connectivity tests to your problem sites.If that checks out you step by step through it. Ask to be moved to a differentaccess router, next change your hardware./Tony-- Tony Sarendal - [EMAIL PROTECTED]IP/Unix -= The scorpion replied, "I couldn't help it, it's my nature" =-
Re: Router / Protocol Problem
Does your peer or you have any ACLs on the PtP link which may be dropping the packets? If your peer is doing uRPF and doesn't have your route properly installed it can cause problems on their edge. Are the sites you cannot reach akamaized? I've had issues with some akamaized sites when I was being redirected to akamai servers that weren't on my network. Do a dig on the website and see if it returns an akamai server Is there any packet loss/CRC errors on the link to your peer? A noisy line will affect large packets more than small packets, I've had issues where only the text/CSS of a website would come up but the images would not. Any MTU issues? Same as above, MTU issues causing large packets to get dropped and no images on websites. Pings, traceroute,telnet all work in those cases -Matt On Sep 6, 2006, at 9:04 AM, Mike Walter wrote: I normally would not post to the group, but I am 100% stumped and have talked with peers with no luck. I have (2) Cisco 7204 Routers running BGP with 3 peers and HSRP. I am not doing anything special with BGP, pretty much a default config that has not changed in years. Recently with no changes to my network, I have been having problems connecting to certain websites and mail servers. I am always able to ping the sites and trace route without error. If I telnet to port 80 or port 25 it does not connect. If I login to my router and telnet sourcing my each of Internet Providers ports, I am able to get to the sites. I have talked with all the providers and none can find a problem. If I shut down one specific peer, everything works fine. So I keep thinking it was that peers problem some how. I have tested with just that peer up and I still can not connect. However, when talking with that peer, they are able to telnet from their network to the sites I can not reach. I don't know what else to check besides shutting down that peer. Which since it is under a 3 year contract, not an option. That isn't the real solution anyhow. Can anyone shed some light on or off-list? Thanks, Mike Walter -- Matthew S. Crocker Vice President Crocker Communications, Inc. Internet Division PO BOX 710 Greenfield, MA 01302-0710 http://www.crocker.com
RE: Router / Protocol Problem
Thanks for everyone's great input. Here are answers to Justin's questions. #1 - 12.3.6a - 7204VXR (NPE400) 512MB - 200+ MB free #2 - 12.2.15T5 - cisco 7204VXR (NPE225) - 256MB (I have a NPE400 - 512MB I want to swap in) - 23MB Free (Issue?) Full Routes from all peers. No internal routing protocol as of yet, all static routes. Getting ready to implement OSPF. I have not rebooted the routers as a test. I have CEF on both routers. I have had some customers complaining about slowness. Mike Walter, MCP Systems Administrator 3z.net a PCD Company http://www.3z.net When Success is the Only Solution think 3z.net Voice (859) 331-9004 Fax (859) 578-3522 -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Justin M. Streiner Sent: Wednesday, September 06, 2006 9:42 AM To: nanog@merit.edu Subject: Re: Router / Protocol Problem On Wed, 6 Sep 2006, Mike Walter wrote: I normally would not post to the group, but I am 100% stumped and have talked with peers with no luck. I have (2) Cisco 7204 Routers running BGP with 3 peers and HSRP. I am not doing anything special with BGP, pretty much a default config that has not changed in years. Please provide details on both your default config and the hardware you're using. You say you have two Cisco 7204s - are these straight '04s, or 7204VXRs? What NPE(s) are you using, and how much memory is on them? The BGP you're getting from your peers - are you getting full routes from any of them? Do you have CEF enabled on these routers? What IOS version(s) are running on these routers? What else are they doing besides slinging BGP routes? Does the problem go away for a while if you reboot one router or the other? Without knowing any of this, it sounds like you might have NPE-225, -300, or -400 with 256 MB of RAM and you are running into memory exhaustion issues from carrying full routes. That's been a pretty popular topic on this list and others like cisco-nsp in the last 12 months :) At a minimum, what do the output of show mem summary and show ip bgp sum from each router show you? Have you seen other performance problems lately, such as things getting mysteriously slower, beyond the rachability issues you mentioned above? If so, check if CEF is still running (if it was configured in the first place). When a 7200 gets dangerously low on free memory and CEF is running, it may cannibalize the IP CEF process to try to conserve memory. Earlier 12.0 releases did this - I don't know if newer ones still do it. jms
RE: Router / Protocol Problem
On Wed, 6 Sep 2006, Mike Walter wrote: Thanks for everyone's great input. Here are answers to Justin's questions. #1 - 12.3.6a - 7204VXR (NPE400) 512MB - 200+ MB free #2 - 12.2.15T5 - cisco 7204VXR (NPE225) - 256MB (I have a NPE400 - 512MB I want to swap in) - 23MB Free (Issue?) Full Routes from all peers. No internal routing protocol as of yet, all static routes. Getting ready to implement OSPF. I have not rebooted the routers as a test. I have CEF on both routers. I have had some customers complaining about slowness. That NPE-225 could be the culprit. 23 MB free is workable, but definitely low. I see that router is running 12.2.15T5. T train IOS releases are where features are normally introduced first and are often more buggy than mainline IOS releases. I wouldn't be surprised if that release has some memory leaks that is exacerbating your issue. If you graph memory utilization with MRTG or some other handy graphing tool, memory leaks often show up as a pronounced 'saw-tooth' pattern when you look at the utilization over time. Also, how badly is the CPU getting hit on that NPE-225? Since you're carrying full routes, I wouldn't be surprised to see the BGP Scanner process beating your CPU up every 60-ish seconds. What does the output of show proc cpu hist show you? If you have lots of spikes up to 100% peak utilization in the history graphs, then that can definitely be a source of latency. jms
RE: Router / Protocol Problem
On Wed, 6 Sep 2006, Mike Walter wrote: Thanks for everyone's great input. Here are answers to Justin's questions. #1 - 12.3.6a - 7204VXR (NPE400) 512MB - 200+ MB free #2 - 12.2.15T5 - cisco 7204VXR (NPE225) - 256MB (I have a NPE400 - 512MB I want to swap in) - 23MB Free (Issue?) Full Routes from all peers. No internal routing protocol as of yet, all static routes. Getting ready to implement OSPF. I have not rebooted the routers as a test. I have CEF on both routers. I have had some customers complaining about slowness. No internal routing protocol? Not even iBGP? How do the 2 routers exchange info? How do the internal systems know which router to exit from? Or are they both independent? I assume you are AS26241 and peer with 3356, 4323 and 6181. I also assume you should be announcing your 2 prefixes: 69.4.64.0/20 216.68.104.0/21 but you have deaggregated a single /24 - 69.4.71.0/24 which has sent 34 BGP updates in the past 24 hours (which might be ok). So, it is a bit hard to debug this with only partial info. Regards, Hank Nussbacher http://www.interall.co.il
RE: Router / Protocol Problem
Sorry, I am running iBGP. I just swapped out the NPE225 engine to a NPE400 and 512MB and have not seen a change yet. I am still unable to reach the sites. I am going to give it a while and sometime soon reboot the other router. I removed the single /24 today out the one connection to see if that would change anything as well. Mike -Original Message- From: Hank Nussbacher [mailto:[EMAIL PROTECTED] Sent: Wednesday, September 06, 2006 12:07 PM To: Mike Walter Cc: Justin M. Streiner; nanog@merit.edu Subject: RE: Router / Protocol Problem On Wed, 6 Sep 2006, Mike Walter wrote: Thanks for everyone's great input. Here are answers to Justin's questions. #1 - 12.3.6a - 7204VXR (NPE400) 512MB - 200+ MB free #2 - 12.2.15T5 - cisco 7204VXR (NPE225) - 256MB (I have a NPE400 - 512MB I want to swap in) - 23MB Free (Issue?) Full Routes from all peers. No internal routing protocol as of yet, all static routes. Getting ready to implement OSPF. I have not rebooted the routers as a test. I have CEF on both routers. I have had some customers complaining about slowness. No internal routing protocol? Not even iBGP? How do the 2 routers exchange info? How do the internal systems know which router to exit from? Or are they both independent? I assume you are AS26241 and peer with 3356, 4323 and 6181. I also assume you should be announcing your 2 prefixes: 69.4.64.0/20 216.68.104.0/21 but you have deaggregated a single /24 - 69.4.71.0/24 which has sent 34 BGP updates in the past 24 hours (which might be ok). So, it is a bit hard to debug this with only partial info. Regards, Hank Nussbacher http://www.interall.co.il
Re: Router / Protocol Problem
Get a sniffer trace. Packets on the wire prove what's going on. Without that kind of real data everything is just speculation. Rodney On Wed, Sep 06, 2006 at 12:16:01PM -0400, Mike Walter wrote: Sorry, I am running iBGP. I just swapped out the NPE225 engine to a NPE400 and 512MB and have not seen a change yet. I am still unable to reach the sites. I am going to give it a while and sometime soon reboot the other router. I removed the single /24 today out the one connection to see if that would change anything as well. Mike -Original Message- From: Hank Nussbacher [mailto:[EMAIL PROTECTED] Sent: Wednesday, September 06, 2006 12:07 PM To: Mike Walter Cc: Justin M. Streiner; nanog@merit.edu Subject: RE: Router / Protocol Problem On Wed, 6 Sep 2006, Mike Walter wrote: Thanks for everyone's great input. Here are answers to Justin's questions. #1 - 12.3.6a - 7204VXR (NPE400) 512MB - 200+ MB free #2 - 12.2.15T5 - cisco 7204VXR (NPE225) - 256MB (I have a NPE400 - 512MB I want to swap in) - 23MB Free (Issue?) Full Routes from all peers. No internal routing protocol as of yet, all static routes. Getting ready to implement OSPF. I have not rebooted the routers as a test. I have CEF on both routers. I have had some customers complaining about slowness. No internal routing protocol? Not even iBGP? How do the 2 routers exchange info? How do the internal systems know which router to exit from? Or are they both independent? I assume you are AS26241 and peer with 3356, 4323 and 6181. I also assume you should be announcing your 2 prefixes: 69.4.64.0/20 216.68.104.0/21 but you have deaggregated a single /24 - 69.4.71.0/24 which has sent 34 BGP updates in the past 24 hours (which might be ok). So, it is a bit hard to debug this with only partial info. Regards, Hank Nussbacher http://www.interall.co.il
Re: Router / Protocol Problem
On Wed, 6 Sep 2006, Rodney Dunn wrote: Get a sniffer trace. Packets on the wire prove what's going on. provided the packets get back to him, it seems his problem is traffic getting back to him :( so probably no packets will be on the wire (none in question atleast)...
Re: Router / Protocol Problem
Then that proves it's not a local router problem then. :) On Wed, Sep 06, 2006 at 07:49:26PM +, Christopher L. Morrow wrote: On Wed, 6 Sep 2006, Rodney Dunn wrote: Get a sniffer trace. Packets on the wire prove what's going on. provided the packets get back to him, it seems his problem is traffic getting back to him :( so probably no packets will be on the wire (none in question atleast)...
