Strange Encryption Problem/Mystery/Bug???? [7:9131]
Hi all, Thought I'd ask if anyone has experienced similar to a problem I've had recently. It's going to TAC now and I've got a workaround, but I'd be interested if anyone could suggest the reason or confirm a bug which I have been unable to locate. These routers had been running for nearly a year with no problems. All of a sudden they started giving problems and I have been unable to track down what external network changes may have triggered it. There were no config changes to the two routers (We run Resource Manager Essentials which monitors router config changes) Set-up was two 3660's running 12.1.1T, DES encryption over serial 2Mb link. The links were set up with encryption peers as the serial IP addresses and encryption access lists set up symmetrically without using any any on the serial interfaces. After reload routers come up fine and encryption runs O.K. After a varying length of time (around 5-10 minutes), the CPU utilisation builds up gradually until it sits at around 98%. This utilisation was almost totally due to encryption process. After swapping out the router and upgrading IOS, the problem still existed. I have got around the problem by using IP Unnumbered Loopback0 on Serial links. CPU utilisation now hovers around 2%. Anybody seen similar, or suggest what could make the CPU utilisation snowball like this? Thanks, Gareth Hinton Message Posted at: http://www.groupstudy.com/form/read.php?f=7&i=9131&t=9131 -- FAQ, list archives, and subscription info: http://www.groupstudy.com/list/cisco.html Report misconduct and Nondisclosure violations to [EMAIL PROTECTED]
Re: Strange Encryption Problem/Mystery/Bug???? [7:9131]
1. Have you checked the health of the serial interfaces? (line hits, CRC errors, dropped packets, etc) 2. Did the traffic over the line increase dramatically? 3. Although it sounds like these are production routers, have you looked at a short-timed trace of 'debug crypto ipsec' and/or 'debug ip isakmp'? Do your 'show crypto' stats show anything revealing? -e- - Original Message - From: Gareth Hinton To: Sent: Tuesday, June 19, 2001 6:29 PM Subject: Strange Encryption Problem/Mystery/Bug [7:9131] > Hi all, > > Thought I'd ask if anyone has experienced similar to a problem I've had > recently. It's going to TAC now and I've got a workaround, but I'd be > interested if anyone could suggest the reason or confirm a bug which I have > been unable to locate. > > These routers had been running for nearly a year with no problems. All of a > sudden they started giving problems and I have been unable to track down > what external network changes may have triggered it. There were no config > changes to the two routers (We run Resource Manager Essentials which > monitors router config changes) > > Set-up was two 3660's running 12.1.1T, DES encryption over serial 2Mb link. > The links were set up with encryption peers as the serial IP addresses and > encryption access lists set up symmetrically without using any any on the > serial interfaces. > > After reload routers come up fine and encryption runs O.K. > After a varying length of time (around 5-10 minutes), the CPU utilisation > builds up gradually until it sits at around 98%. This utilisation was almost > totally due to encryption process. > > After swapping out the router and upgrading IOS, the problem still existed. > > I have got around the problem by using IP Unnumbered Loopback0 on Serial > links. CPU utilisation now hovers around 2%. > > Anybody seen similar, or suggest what could make the CPU utilisation > snowball like this? > > > Thanks, > > Gareth Hinton Message Posted at: http://www.groupstudy.com/form/read.php?f=7&i=9145&t=9131 -- FAQ, list archives, and subscription info: http://www.groupstudy.com/list/cisco.html Report misconduct and Nondisclosure violations to [EMAIL PROTECTED]
Re: Strange Encryption Problem/Mystery/Bug???? [7:9131]
Since you've already replaced the router and upgraded the IOS, my first guess is that this is traffic related. Perhaps you have a routing loop or something of that nature. Even though it might take your router down, have you tried debug ip packet?Seriously, this might help pinpoint the offenders. Another thought is to just look at devices that are generating traffic that is being encrypted. I know, that's obvious, but that's what I would do. It may not be a routing loop, per say, it may just be a crazy device generating a lot of traffic that is hitting your crypto access lists. Being a slightly crazy person, I'm still leaning toward doing some debugging. :-) There may be some other crypto-related debugs that point out what's happening, I'll have to check into that when I get back to work tomorrow. Oh, another thought. Do a 'show crypto ipsec sa' and look for unusually high counters. That might also lead you to the culprit(s). Good luck, and let us know if you find anything. John | Hi all, | | Thought I'd ask if anyone has experienced similar to a problem I've had | recently. It's going to TAC now and I've got a workaround, but I'd be | interested if anyone could suggest the reason or confirm a bug which I have | been unable to locate. | | These routers had been running for nearly a year with no problems. All of a | sudden they started giving problems and I have been unable to track down | what external network changes may have triggered it. There were no config | changes to the two routers (We run Resource Manager Essentials which | monitors router config changes) | | Set-up was two 3660's running 12.1.1T, DES encryption over serial 2Mb link. | The links were set up with encryption peers as the serial IP addresses and | encryption access lists set up symmetrically without using any any on the | serial interfaces. | | After reload routers come up fine and encryption runs O.K. | After a varying length of time (around 5-10 minutes), the CPU utilisation | builds up gradually until it sits at around 98%. This utilisation was almost | totally due to encryption process. | | After swapping out the router and upgrading IOS, the problem still existed. | | I have got around the problem by using IP Unnumbered Loopback0 on Serial | links. CPU utilisation now hovers around 2%. | | Anybody seen similar, or suggest what could make the CPU utilisation | snowball like this? | | | Thanks, | | Gareth Hinton | | | | ___ Send a cool gift with your E-Card http://www.bluemountain.com/giftcenter/ Message Posted at: http://www.groupstudy.com/form/read.php?f=7&i=9140&t=9131 -- FAQ, list archives, and subscription info: http://www.groupstudy.com/list/cisco.html Report misconduct and Nondisclosure violations to [EMAIL PROTECTED]
Re: Strange Encryption Problem/Mystery/Bug???? [7:9131]
Thanks for replies, Checked the health of the serial lines. They showed nothing at all, not one error. There was an increase in traffic on the serial interfaces, but not on any other. Jim Gillen suggested a routing loop and it could be the case that something is bouncing back and forth between the interfaces. I've yet to sift through the two routing tables. Couldn't see anything out of the ordinary in the debug crypto ipsec, but have to admit my debug knowledge on encryption is lacking and didn't try debug ip isakmp. Before we had to put the workaround in, we managed to get a debug ip packet on the serial interface which showed a fair bit of traffic between the two serial addresses. It's running in transport mode, so I take it the only traffic between these interfaces is the two routers talking to each other (rather than encapsulated traffic) Regards, Gareth ""EA Louie"" wrote in message [EMAIL PROTECTED]">news:[EMAIL PROTECTED]... > 1. Have you checked the health of the serial interfaces? (line hits, CRC > errors, dropped packets, etc) > 2. Did the traffic over the line increase dramatically? > 3. Although it sounds like these are production routers, have you looked at > a short-timed trace of 'debug crypto ipsec' and/or 'debug ip isakmp'? Do > your 'show crypto' stats show anything revealing? > > -e- > > - Original Message ----- > From: Gareth Hinton > To: > Sent: Tuesday, June 19, 2001 6:29 PM > Subject: Strange Encryption Problem/Mystery/Bug [7:9131] > > > > Hi all, > > > > Thought I'd ask if anyone has experienced similar to a problem I've had > > recently. It's going to TAC now and I've got a workaround, but I'd be > > interested if anyone could suggest the reason or confirm a bug which I > have > > been unable to locate. > > > > These routers had been running for nearly a year with no problems. All of > a > > sudden they started giving problems and I have been unable to track down > > what external network changes may have triggered it. There were no config > > changes to the two routers (We run Resource Manager Essentials which > > monitors router config changes) > > > > Set-up was two 3660's running 12.1.1T, DES encryption over serial 2Mb > link. > > The links were set up with encryption peers as the serial IP addresses and > > encryption access lists set up symmetrically without using any any on the > > serial interfaces. > > > > After reload routers come up fine and encryption runs O.K. > > After a varying length of time (around 5-10 minutes), the CPU utilisation > > builds up gradually until it sits at around 98%. This utilisation was > almost > > totally due to encryption process. > > > > After swapping out the router and upgrading IOS, the problem still > existed. > > > > I have got around the problem by using IP Unnumbered Loopback0 on Serial > > links. CPU utilisation now hovers around 2%. > > > > Anybody seen similar, or suggest what could make the CPU utilisation > > snowball like this? > > > > > > Thanks, > > > > Gareth Hinton Message Posted at: http://www.groupstudy.com/form/read.php?f=7&i=9163&t=9131 -- FAQ, list archives, and subscription info: http://www.groupstudy.com/list/cisco.html Report misconduct and Nondisclosure violations to [EMAIL PROTECTED]
