Hi, I've got something really interesting to show, which shows this
clearly and should help point to the root cause.
In short, it seems that the desired nexthop is not applied by the CARP
master when it is in state 'nexthop 180.25.32.20 now valid: via
180.25.32.20'. I.e. when it is 'via' even though it is a local IP..
From the perspective of the 'backup' the CARP IP is a directly
connected IP which it can reach 'nexthop 180.25.32.20 now valid:
directly connected'.
NB; I haven't had a chance to test IPv6 or iBGP but from this
observation it looks like the same problem will be seen, unless there
is a way of telling OpenBGPd to use nexthops which are 'via' something..
THE SETUP;
- Two OpenBSD boxes with CARP on their BGP and LAN Interfaces.
- One or two upstream Cisco routers on BGP interface via switch (both
show same problem).
- PF disabled (just for this testing).
- 180.25.32.1 = iBGP Cisco Router
- 180.25.32.20 = CARP IP
- 180.25.32.21 = OBSD1
- 180.25.32.22 = OBSD2
- Neighbors are eBGP
OpenBSD Host 1 (master) /etc/bgpd.conf;
AS 66868
router-id 180.25.32.21
log updates
network 180.25.32.0/22
network 2a00:7ee0::/32
neighbor 180.25.32.1 {
remote-as 66868
announce self
local-address 180.25.32.21
tcp md5sig password secret
descr "THN"
}
match to 180.25.32.1 set nexthop 180.25.32.20
allow from any inet prefixlen 8 - 26
allow from any inet6 prefixlen 16 - 48
allow to any
OpenBSD Host 1 (backup) /etc/bgpd.conf;
AS 66868
router-id 180.25.32.22
log updates
network 180.25.32.0/22
network 2a00:7ee0::/32
neighbor 180.25.32.1 {
remote-as 66868
announce self
local-address 180.25.32.22
tcp md5sig password secret
descr "THN"
}
match to 180.25.32.1 set nexthop 180.25.32.20
allow from any inet prefixlen 8 - 26
allow from any inet6 prefixlen 16 - 48
allow to any
Cisco Host;
router bgp 12345
bgp router-id 180.25.32.1
bgp log-neighbor-changes
neighbor 180.25.32.21 remote-as 66868
neighbor 180.25.32.21 password secret
neighbor 180.25.32.22 remote-as 66868
neighbor 180.25.32.22 password secret
!
address-family ipv4
neighbor 180.25.32.21 activate
neighbor 180.25.32.22 activate
exit-address-family
!
!
TEST 1 - Start BGP on master then backup;
BGP Process is already running on the Cisco..
THN(config)#do show ip bgp
THN(config)#
OpenBSD Host 1 (MASTER) bgpd -dv;
[LIVE]root@OpenBSD1:~# bgpd -dv
startup
rereading config
route decision engine ready
session engine ready
new ktable rdomain_0 for rtableid 0
nexthop 180.25.32.20 now valid: via 180.25.32.20
listening on 0.0.0.0
listening on ::
SE reconfigured
neighbor 180.25.32.1 (THN): state change None -> Idle, reason: None
neighbor 180.25.32.1 (THN): state change Idle -> Connect, reason: Start
RDE reconfigured
neighbor 180.25.32.1 (THN): state change Connect -> OpenSent, reason:
Connection opened
neighbor 180.25.32.1 (THN): state change OpenSent -> OpenConfirm,
reason: OPEN message received
neighbor 180.25.32.1 (THN): state change OpenConfirm -> Established,
reason: KEEPALIVE message received
THN(config)#do show ip bgp
BGP table version is 8, local router ID is 180.25.32.1
Status codes: s suppressed, d damped, h history, * valid, > best, i -
internal,
r RIB-failure, S Stale, m multipath, b backup-path, f
RT-Filter,
x best-external, a additional-path, c RIB-compressed,
Origin codes: i - IGP, e - EGP, ? - incomplete
RPKI validation codes: V valid, I invalid, N Not found
Network Next Hop Metric LocPrf Weight Path
*> 180.25.32.0/22 180.25.32.21 0 66868 i
NOTICE THIS IS THE WRONG NEXTHOP! :(
OpenBSD Host 2 (BACKUP) bgpd -dv;
[LIVE]root@OpenBSD2:~# bgpd -dv
startup
rereading config
route decision engine ready
session engine ready
new ktable rdomain_0 for rtableid 0
nexthop 180.25.32.20 now valid: directly connected
listening on 0.0.0.0
listening on ::
SE reconfigured
neighbor 180.25.32.1 (THN): state change None -> Idle, reason: None
neighbor 180.25.32.1 (THN): state change Idle -> Connect, reason: Start
RDE reconfigured
neighbor 180.25.32.1 (THN): state change Connect -> OpenSent, reason:
Connection opened
neighbor 180.25.32.1 (THN): state change OpenSent -> OpenConfirm,
reason: OPEN message received
neighbor 180.25.32.1 (THN): state change OpenConfirm -> Established,
reason: KEEPALIVE message received
Rib Loc-RIB: neighbor 180.25.32.1 (THN) AS12345: update 180.25.32.0/22
via 180.25.32.1
nexthop 180.25.32.1 now valid: directly connected
(^ Why do these last two lines not show on the master?)
THN(config)#do show ip bgp
BGP table version is 8, local router ID is 180.25.32.1
Status codes: s suppressed, d damped, h history, * valid, > best, i -
internal,
r RIB-failure, S Stale, m multipath, b backup-path, f
RT-Filter,
x best-external, a additional-path, c RIB-compressed,
Origin codes: i - IGP, e - EGP, ? - incomplete
RPKI validation codes: V valid, I invalid, N Not found
Network Next Hop Metric LocPrf Weight Path
* 180.25.32.0/22 180.25.32.20 0 66868 i
*> 180.25.32.21 0 66868 i
THE CORRECT NEXTHOP IS SHOWN (180.25.32.20) BUT IS NOT THE > BEST
TEST 2 - Shutdown OpenBGPd on both and restart OpenBGPd on just the
backup;
OpenBSD Host 2 (BACKUP) bgpd -dv;
[LIVE]root@OpenBSD2:~# bgpd -dv
startup
rereading config
route decision engine ready
session engine ready
new ktable rdomain_0 for rtableid 0
nexthop 180.25.32.20 now valid: directly connected
listening on 0.0.0.0
listening on ::
SE reconfigured
neighbor 180.25.32.1 (THN): state change None -> Idle, reason: None
neighbor 180.25.32.1 (THN): state change Idle -> Connect, reason: Start
RDE reconfigured
neighbor 180.25.32.1 (THN): state change Connect -> OpenSent, reason:
Connection opened
neighbor 180.25.32.1 (THN): state change OpenSent -> OpenConfirm,
reason: OPEN message received
neighbor 180.25.32.1 (THN): state change OpenConfirm -> Established,
reason: KEEPALIVE message received
THN(config)#do show ip bgp
BGP table version is 14, local router ID is 180.25.32.1
Status codes: s suppressed, d damped, h history, * valid, > best, i -
internal,
r RIB-failure, S Stale, m multipath, b backup-path, f
RT-Filter,
x best-external, a additional-path, c RIB-compressed,
Origin codes: i - IGP, e - EGP, ? - incomplete
RPKI validation codes: V valid, I invalid, N Not found
Network Next Hop Metric LocPrf Weight Path
*> 180.25.32.0/22 180.25.32.20 0 66868 i
THE CORRECT NEXTHOP IS STILL SHOWN AND OF COURSE IS NOW THE BEST AS ITS
THE ONLY ONE..
TEST 3 - Now lets start OpenBGPd on the master;
[LIVE]root@OpenBSD1:~# bgpd -dv
startup
rereading config
route decision engine ready
session engine ready
new ktable rdomain_0 for rtableid 0
nexthop 180.25.32.20 now valid: via 180.25.32.20
listening on 0.0.0.0
listening on ::
SE reconfigured
neighbor 180.25.32.1 (THN): state change None -> Idle, reason: None
neighbor 180.25.32.1 (THN): state change Idle -> Connect, reason: Start
RDE reconfigured
neighbor 180.25.32.1 (THN): state change Connect -> OpenSent, reason:
Connection opened
neighbor 180.25.32.1 (THN): state change OpenSent -> OpenConfirm,
reason: OPEN message received
neighbor 180.25.32.1 (THN): state change OpenConfirm -> Established,
reason: KEEPALIVE message received
Rib Loc-RIB: neighbor 180.25.32.1 (THN) AS12345: update 180.25.32.0/22
via 180.25.32.1
nexthop 180.25.32.1 now valid: directly connected
THN(config)#do show ip bgp
BGP table version is 14, local router ID is 180.25.32.1
Status codes: s suppressed, d damped, h history, * valid, > best, i -
internal,
r RIB-failure, S Stale, m multipath, b backup-path, f
RT-Filter,
x best-external, a additional-path, c RIB-compressed,
Origin codes: i - IGP, e - EGP, ? - incomplete
RPKI validation codes: V valid, I invalid, N Not found
Network Next Hop Metric LocPrf Weight Path
* 180.25.32.0/22 180.25.32.21 0 66868 i
*> 180.25.32.20 0 66868 i
THE MASTER IS STILL SENDING A NEXTHOP OF ITS PHYSICAL INTERFACE AND NOT
THE CARP IP, SO THE STARTING ORDER DOESN'T MATTER AND THIS ISN'T SOME
ROUTE REFLECTION WIERDNESS
TEST 4 - Now lets stop OpenBGPd on the master, switch the firewalls to
make the master the backup and restart OpenBGPd;
[LIVE]root@OpenBSD1:~# ifconfig -g carp carpdemote 10
This following message appeared on the OpenBGPd debug on the backup as
I performed the carpdemote on the master;
nexthop 180.25.32.20 now valid: via 180.25.32.20
[LIVE]root@OpenBSD1:~# bgpd -dv
startup
rereading config
route decision engine ready
session engine ready
new ktable rdomain_0 for rtableid 0
nexthop 180.25.32.20 now valid: directly connected
listening on 0.0.0.0
listening on ::
SE reconfigured
neighbor 180.25.32.1 (THN): state change None -> Idle, reason: None
neighbor 180.25.32.1 (THN): state change Idle -> Connect, reason: Start
RDE reconfigured
neighbor 180.25.32.1 (THN): state change Connect -> OpenSent, reason:
Connection opened
neighbor 180.25.32.1 (THN): state change OpenSent -> OpenConfirm,
reason: OPEN message received
neighbor 180.25.32.1 (THN): state change OpenConfirm -> Established,
reason: KEEPALIVE message received
Rib Loc-RIB: neighbor 180.25.32.1 (THN) AS12345: update 180.25.32.0/22
via 180.25.32.1
nexthop 180.25.32.1 now valid: directly connected
THN#show ip bgp
BGP table version is 14, local router ID is 180.25.32.1
Status codes: s suppressed, d damped, h history, * valid, > best, i -
internal,
r RIB-failure, S Stale, m multipath, b backup-path, f
RT-Filter,
x best-external, a additional-path, c RIB-compressed,
Origin codes: i - IGP, e - EGP, ? - incomplete
RPKI validation codes: V valid, I invalid, N Not found
Network Next Hop Metric LocPrf Weight Path
* 180.25.32.0/22 180.25.32.20 0 66868 i
*> 180.25.32.20 0 66868 i
WE NOW HAVE TWO ROUTES IN THE CISCO BGP RIB WITH THE CARP IP AS A
RESULT OF ONLY STARTING OPENBGPD WHEN THE FIREWALL IS A BACKUP.
TEST 5 - Without shutting down OpenBGPd switch the firewalls back
[LIVE]root@OpenBSD1:~# ifconfig -g carp -carpdemote 10
[LIVE]root@OpenBSD1(debug);
nexthop 180.25.32.1 now valid: directly connected
nexthop 180.25.32.20 now valid: via 180.25.32.20
[LIVE]root@OpenBSD2(debug);
nexthop 180.25.32.20 now valid: via 180.25.32.20
nexthop 180.25.32.20 now valid: directly connected
THN#show ip bgp
BGP table version is 14, local router ID is 180.25.32.1
Status codes: s suppressed, d damped, h history, * valid, > best, i -
internal,
r RIB-failure, S Stale, m multipath, b backup-path, f
RT-Filter,
x best-external, a additional-path, c RIB-compressed,
Origin codes: i - IGP, e - EGP, ? - incomplete
RPKI validation codes: V valid, I invalid, N Not found
Network Next Hop Metric LocPrf Weight Path
* 180.25.32.0/22 180.25.32.20 0 66868 i
*> 180.25.32.20 0 66868 i
THE ROUTES CONTINUE TO BE ANNOUNCED USING THE CARP IP.
So we can see that the nexthop is only correctly set when; nexthop
180.25.32.20 now valid: directly connected
On Tue 03 Dec 2013 02:26:30 GMT, athom...@athompso.net wrote:
No, I'm seeing the same thing - the carp master advertises the carp IP as
next-hop no matter what.
The carp backup advertises whatever you've told it to advertise via "set
nexthop".
-Adam
On Dec 2, 2013 6:43 PM, Chris Cappuccio <ch...@nmedia.net> wrote:
andy [a...@brandwatch.com] wrote:
Hi,
Could someone help me with this issue we have found where the OpenBGPd
rule 'match to bgppeerip set nexthop bgpcarpip' doesn't work if OpenBGPd is
started whilst the OpenBSD host is a carp master. It only works if it is a
CARP backup :(
Or could someone give me a clue where in the source code to look so I can
try to comment out the code which is checking the state of carp? This is
desperately important for us for reasons discussed in this thread and
others.
Thanks for your time, Andy.
PS; Thanks to Henning and Claudio for their great work with OpenBGPd.
Can you demonstrate the failure through any bgpd output or some other way?
For instance, does bgpd fail to advertise routes via bgp if it's the CARP
nexthop master?
Or does it all look like it should work, and just fail?
