A "show memory allocating-process totals" is very telling.
llocator PC Summary for: Processor
Displayed first 2048 Allocator PCs only
PC Total Count Name
0x4035A6C8 295186172 4420 BGP battr chun
0x40809510 45688268 695 CEF: fib
BGP is definitely the culprit. It has consumed almost 250 million more
bytes than the next closest process. Yikes!
Looking at a "healthy" router, this process is just under the 100
million byte mark.
I received an off-list reply that contained the workaround. I had four
BGP sessions in an admin down state and one that was trying to connect.
I removed all five of these sessions from my configuration and the
difference was dramatic.
Here's the path/bestpath before the removal of the configuration:
1640389/55395 BGP path/bestpath attribute entries using 262462240 bytes
of memory
Here it is after:
463745/55388 BGP path/bestpath attribute entries using 74199200 bytes of
memory
That's a staggering difference.
However, while the memory has been released back into the "BGP memory
pool", it does not show up in the "free memory pool". We're still at
90% usage, so I will have to proceed with our scheduled maintenance tonight.
I had planned on moving to SXI2 tonight, but it sounds like that has
some memory issues as well. Think I might just stay put for now, since
I now know the workaround for this issue.
Thank you everyone for your replies and assistance. It was of great help!
Cheers!
e ninja wrote:
Grab multiple captures of sh proc mem to identify the process "holding"
and not releasing (i.e. leaking) memory. When memory is heavily
depleted, grab a *show memory allocating-process totals* and feel free
to unicast.
Any MALLOC failures?
-Eninja
On Sun, Aug 30, 2009 at 10:11 AM, Chris Phillips
<cphill...@wbsconnect.com <mailto:cphill...@wbsconnect.com>> wrote:
Every six weeks or so I am running out of memory on a 6509 w/ dual
SUP720-3BXL with mostly 6700-series line cards.
I have 21 other nodes with this exact same configuration, some even
running SXI or SXI1 that do not have this issue, which first led me
to believe that the issue might be hardware related.
During our last maintenance window to alleviate the memory issue, I
forced the standby SUP to become the active SUP. The memory issue
persisted, leading me back to thinking it is a software issue.
I did not have this issue with SXH* on this same device, but SXH is
*SO* buggy, rolling back is not an option. This leads me to believe
that it is most likely a software issue.
The router is heavily used with 250+ BGP sessions, OSPF, MPLS,
v4/v6, etc, but I don't think it should be consuming and not
releasing 4 mbytes of memory each day.
Has anyone else seen this? Anyone know a workaround?
I'm upgrading to SXI2 tomight in hopes that it resolves my issue.
--
Chris Phillips
_______________________________________________
cisco-nsp mailing list cisco-nsp@puck.nether.net
<mailto:cisco-nsp@puck.nether.net>
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/
--
Chris Phillips
_______________________________________________
cisco-nsp mailing list cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/
