I guess I should start from the beginning. When I originally started this
project my goal was to have two machines running carp between them and have
the master connect to two different ISP's sending full routes. This was
working fine and failover didn't cause any issues. At least I thought
everything was fine until I rebooted the machine. For a reason I could not
figure out, I would end up having a kernel panic if I didn't kill bgpd at
startup. What was happening would be the machine would boot, become master
of the carp interfaces, bgpd would connect to ebgp peers and fill the rib
and then process and add to fib. After this occured the machine would look
to itself and totally ignore the fib. While this goes on the host is not
having a problem receiving updates from it's peers right up until the kernel
panics. I even built 4.1 just to test and had the same problem. I didn't
have enough time to keep investigating so I plan on getting dumps to post
and hopefully aid with tracking down this problem, if it truly is one.
With that out of the way I can explain the problem I'm having now. Since I
was unable to get the previous solution working I decided to split the
peering between the two servers, not have bgpd rely on carp and just use
carp for my gateway. With this in place I started workign on ibgp between
the boxes to make sure that no matter where my default route went, the host
would send me through the best provider be it directly attached or off of
the other host. Right off the bat I started noticing a problem that I
thought was either a configuration error or a misunderstanding of IBGP.
When the hosts connect to each other a full prefix table is sent but then
almost immediately it's neighbor starts withdrawing prefixes. On one host I
jump from 210k prefixes in the initial connection to only 59k after all of
the withdrawals. On the other host I go from 210k to 197k which isn't as
bad but I'm still unsure of why it does this. I started to think that if
IBGP advertises a prefix that the other host is trying to advertise as well,
it just withdraws it from the neighbor since it already is announcing it but
the numbers just don't add up. I came to this conclusion because if I have
one of the hosts announce none, the IBGP neighbor doesn't start withdrawing
after it sends the entire prefix table. Hopefully someone has some ideas.
I also tried to convert ibgp to route-reflectors but I don't think removing
'announce all' and adding 'route-reflector' did anything because I saw the
same behavior.
Below is my config:
Host1
AS 111
router-id 172.16.1.1
listen on 172.16.1.1
listen on 172.16.2.1
network 192.168.10.0/24
network 192.168.11.0/24
neighbor 172.16.1.2 {
remote-as 6517
descr yipes-1
local-address 172.16.1.1
holdtime 180
announce self
depend on trunk0
}
neighbor 172.16.1.3 {
remote-as 6517
descr yipes-2
local-address 172.16.1.1
holdtime 180
announce self
depend on trunk0
}
neighbor 172.16.2.2 {
remote-as 111
descr dyno1
local-address 172.16.2.1
holdtime 180
announce all
depend on trunk0
set nexthop self
}
host2
AS 111
router-id 172.16.3.1
listen on 172.16.3.1
listen on 172.16.2.2
network 192.168.10.0/24
network 192.168.11.0/24
neighbor 172.16.3.2 {
remote-as 174
descr cogent-1
local-address 172.16.3.1
holdtime 180
announce self
depend on trunk0
}
neighbor 172.16.2.1 {
remote-as 111
descr dyno2
local-address 172.16.2.2
holdtime 180
announce all
depend on trunk0
set nexthop self
}
Anyone have any ideas? Thanks for taking the time to read this.
