Hi,
I just fell over this thread while doing a little reseach to solve a
similar situation.
Hardware:
- 6509 with SUP720-3BXL on both ends
- SXF15a
- Uptime: 46 weeks
Problem:
- OSPF (for the loopback between cores) and BGP (mostly customers
whom we send the full table) going up and down all the time:
%OSPF-5-ADJCHG: Process 1, Nbr x.x.x.130 on TenGigabitEthernet4/1
from FULL to DOWN, Neighbor Down: Dead timer expired
%OSPF-5-ADJCHG: Process 1, Nbr x.x.x.131 on TenGigabitEthernet9/1
from LOADING to FULL, Loading Done
%BGP-5-ADJCHANGE: neighbor y.y.y.14 Down BGP Notification sent
%BGP-3-NOTIFICATION: sent to neighbor y.y.y.14 4/0 (hold time
expired) 0 bytes %BGP-5-ADJCHANGE: neighbor y.y.y.14 Up
This keeps going on for several hours, and suddenly it stabilizes
itself.
Furthermore I use cacti to generate graphs from the core router via
SNMP. I have one VLAN that has around 15 GBPS traffic at peak times,
and as soon as I hit more than 15 GBPS, no more graphs are drawn,
core router console becomes rather unresponsive and OSPF starts to
behave strangely.
What I can rule out is the fiber capacity. I have multiple circuits
and different paths and operators. The OSPF issue happens on all
circuits, not just a specific one. No 10 GE link is used more than
60%. In fact, traffic from inside my backbone to any place outside
remains unaffected (thank God), but the core router itself is pretty
useless. Pinging the core's loopback or any ip loaded on that box
results in a 40-60% packet loss.
CPU usage is not high, it's stable. No unusual processes, just IP
Input and BGP Scanner. More than 50% memory is still free at that
time.
I've had this many times recently, but it really just happens when
my core goes beyond +- 15 GBPS of traffic (outbound). We've been
below 15 GBPS for 2 years and it never happaned at that time. Now
all this mess happens almost daily, rendering important billing
graphs useless and annoying full table BGP customers.
Is this a memory issue, due to the router's long uptime? Would
reloading the router help in this case? That's the last thing I
would want to do, but if it helps...
Cheers,
Andy
On Fri, Dec 11, 2009 at 5:22 PM, Drew Weaver
<drew.wea...@thenap.com> wrote:
Howdy all,
Last night I had an interesting encounter on one of my 6509s /w
SUP7203-BXL.
This switch has 3x iBGP sessions with full internet tables and is
also running OSPF.
Two of the three iBGP sessions randomly dropped with:
%BGP-3-NOTIFICATION: sent to neighbor x.x.x.3 4/0 (hold time
expired) 0 bytes, I also noticed that during this period OSPF
dropped with Neighbor Down: Dead timer expired
and then re-established, and then failed again, and
re-established, and failed again, and so-on, and so-on.
I checked the physical interfaces between this 6500 and the two
GSR 12000s it peers with and there were no errors, there was also
no obvious spike in traffic that would account for latency that
might cause the hold timers to expire. I remember when this system
first came online it took a really long time for it to download
the full internet tables from the upstream GSRs and also during
that time there was a lot of CPU time being eaten up, I am
wondering if maybe the first session failing caused sort of a
'performance' domino effect which then caused everything else to
fail, the issue eventually corrected itself and stabilized.
This particular box is running 12.2(18)SXF17 so I am less likely
to believe it is a software bug.
Does anyone have any tips on both how I can avoid the hold timer
issue altogether and also how I can make it so that if a session
does go down and re-establish it doesn't totally nail the CPU
while it's trying to re-establish/download the routes? A long time
ago I also read that increasing the MTU on both ends of a circuit
can make BGP tables download faster, I don't know if that's true
or not, has anyone else found that?
thanks,
-Drew
_______________________________________________
cisco-nsp mailing list cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/
_______________________________________________
cisco-nsp mailing list cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/
