Re: Failover routers with OpenBGPD and independent BGP sessions
X Y wrote :
I'm having a bit of trouble with the finer details of my OpenBGPD
config, and would appreciate some tips on getting it right and advice
on the right way of doing things.
I have two routers, two independent BGP connections, and a block of
provider independent address space. The routers are arranged in a
redundant pair. The public network and some private subnets have
gateway addresses provided with CARP. The two routers use pfsync.
The BGP connections are actually completely independent (I'll be
adding two more in due course for a total of four). They have
different network addresses, cables and route to the rest of the
world. The cables are plugged directly into the routers, and there's
no CARP on those interfaces. Packets will arrive via either of those
routes.
I have got a basic configuration working. This maintains the BGP
sessions, packets go in and out, and the firewalls will fail over as
they should. I use depend on carp0 ... carp3 on the master router
(chosen via advskew) to drop that session if it fails, and demote on
the backup to make sure it doesn't like being master if it doesn't
have a BGP session.
I have been recommended by our ISPs that I should also advertise
routes between the routers, so that if one's BGP session fails, it can
route packets to the other for a cleaner failover. I have not managed
to get this configuration working.
Some configuration information, with the real details removed to
protect the guilty.
AS: 9
PI subnet: A.A.A.0/23
PI gateway: A.A.A.1
Master: A.A.A.2
Backup: A.A.A.3
BGP connection 1: X.X.X.4 - X.X.X.200 on X.X.X.0/24, AS 8
BGP connection 2: Y.Y.Y.4 - Y.Y.Y.200 on Y.Y.Y.0/24, AS 8
(Y.Y.Y !=3D X.X.X)
/etc/bgpd.conf
AS 9
network A.A.A.0/23
neighbor X.X.X.200 {
remote-as 8
local-address X.X.X.4
announce self
tcp md5sig password PASSWORD1
depend on carp1
depend on carp2
depend on carp3
# demote on backup
}
neighbor A.A.A.3 {
remote-as 9
descr backup
local-address A.A.A.2
announce all
tcp md5sig password PASSWORD2
set nexthop A.A.A.3 # A.A.A.2 didn't help
set localpref -10
}
Then...
Stuart Henderson [EMAIL PROTECTED] wrote:
On 2006/10/13 11:24, Ronnie Garcia wrote:
I have been recommended by our ISPs that I should also advertise
routes between the routers, so that if one's BGP session fails, =20
it can
route packets to the other for a cleaner failover. I have not =20
managed
to get this configuration working.
Yes you should, this is called iBGP. All of your BGP routers =20
should have
a iBGP session with all of the others, in a full mesh (unless you are
using a route reflector).
OP has already done that in the config file, the problem is how to
add a route so the other provider's router can be reached. Normally =20=
the
provider's router is listed in the IBGP announcement so unless this is
overwritten in the IBGP announcements (by 'set nexthop) you need to =20=
have
a route to the provider's router (static or OSPF).
I think this is the critical bit of information. I need to add a =20
static route to the other router for the X.X.X/24 or Y.Y=10.Y/24 =20
network. The IBGP session from the other router will give it's =20
neighbour's address, not it's own address which I had expected. I =20
can't override this with nexthop, I just have to make sure there are =20
routes provided via something other than BGP.
A couple of follow-up questions:
1) Will the set localpref -10 on the session with the other router =20
be sufficient to make sure that when the main BGP session is up, =20
that's actually used?
2) When I get to use multiple locations, should I use ospfd rather =20
than BGP to manage which route to take internally to the network?
Thanks for the help,
Ben
Re: Failover routers with OpenBGPD and independent BGP sessions
On 2006/10/19 17:57, X Y wrote: I can't override this with nexthop are you sure? this should work. you are setting it on the _sending_ machine and not the _receiving_ machine aren't you? looking at `bgpd -nv' may help rtr2$ bgpctl sh ip bgp x.x.0.0 flags: * = Valid, = Selected, I = via IBGP, A = Announced origin: i = IGP, e = EGP, ? = Incomplete flags destination gateway lpref med aspath origin I x.x.0/22y.y.187.61 100 0 blah blah blah i rtr1$ sudo vi /etc/bgpd.conf (add 'set nexthop self') rtr1$ bgpctl reload wait for routes to feed across rtr2$ bgpctl sh ip bgp x.x.0.0 flags: * = Valid, = Selected, I = via IBGP, A = Announced origin: i = IGP, e = EGP, ? = Incomplete flags destination gateway lpref med aspath origin I x.x.0/22y.y.187.35 100 0 blah blah blah i 1) Will the set localpref -10 on the session with the other router be sufficient to make sure that when the main BGP session is up, that's actually used? yes, localpref overrides everything else (*including* AS path length). http://unduli.bsws.de/papers/linuxforum2006/mgp00016.txt 1. check if prefix is eligible a.k.a reachable 2. localpref, bigger is better 3. aspath length, the shorter the better 4. origin, the lower the better 5. MED decision, only comparable between the same neighboring AS 6. EBGP is cooler than IBGP 7. weight, bigger is better (extension) 8. route age: older is better (extension, off by default) 9. lowest BGP ID wins 10. lowest peer address wins most of it is standard BGP but the extensions aren't, this list should probably be added to bgpd(8) or possibly bgp.conf(5)... (if anyone would like to express a preference as to which manpage I can prepare a diff) 2) When I get to use multiple locations, should I use ospfd rather than BGP to manage which route to take internally to the network? I assume you are talking about using it to tell which route to take from the border to your internal networks (hosted machines and so on)? that's up to you :-)
Re: Failover routers with OpenBGPD and independent BGP sessions
X Y a icrit :
I have two routers, two independent BGP connections, and a block of
provider independent address space. The routers are arranged in a
redundant pair. The public network and some private subnets have
gateway addresses provided with CARP. The two routers use pfsync.
The BGP connections are actually completely independent (I'll be
adding two more in due course for a total of four). They have
different network addresses, cables and route to the rest of the
world. The cables are plugged directly into the routers, and there's
no CARP on those interfaces. Packets will arrive via either of those
routes.
I have got a basic configuration working. This maintains the BGP
sessions, packets go in and out, and the firewalls will fail over as
they should. I use depend on carp0 ... carp3 on the master router
(chosen via advskew) to drop that session if it fails, and demote on
the backup to make sure it doesn't like being master if it doesn't
have a BGP session.
I have been recommended by our ISPs that I should also advertise
routes between the routers, so that if one's BGP session fails, it can
route packets to the other for a cleaner failover. I have not managed
to get this configuration working.
Yes you should, this is called iBGP. All of your BGP routers should have
a iBGP session with all of the others, in a full mesh (unless you are
using a route reflector).
In your design, you will then get the best routes on each of your border
routers.
Some configuration information, with the real details removed to
protect the guilty.
AS: 9
PI subnet: A.A.A.0/23
PI gateway: A.A.A.1
Master: A.A.A.2
Backup: A.A.A.3
BGP connection 1: X.X.X.4 - X.X.X.200 on X.X.X.0/24, AS 8
BGP connection 2: Y.Y.Y.4 - Y.Y.Y.200 on Y.Y.Y.0/24, AS 8
(Y.Y.Y != X.X.X)
[...]
neighbor A.A.A.3 {
remote-as 9
descr backup
local-address A.A.A.2
announce all
tcp md5sig password PASSWORD2
set nexthop A.A.A.3# A.A.A.2 didn't help
set localpref -10
}
You shouldn't need a nexthop here.
In iBGP sessions, you should set the neighbor address to be the loopback
address of your other border router.
Your router-id parameter should also be the IP adress of your local
loopback interface.
Your loopback interfaces should have a /32 IP adress set.
Regards,
--
Ronnie Garcia r.garcia at ovea dot com
Re: Failover routers with OpenBGPD and independent BGP sessions
On 2006/10/13 11:24, Ronnie Garcia wrote: I have been recommended by our ISPs that I should also advertise routes between the routers, so that if one's BGP session fails, it can route packets to the other for a cleaner failover. I have not managed to get this configuration working. Yes you should, this is called iBGP. All of your BGP routers should have a iBGP session with all of the others, in a full mesh (unless you are using a route reflector). OP has already done that in the config file, the problem is how to add a route so the other provider's router can be reached. Normally the provider's router is listed in the IBGP announcement so unless this is overwritten in the IBGP announcements (by 'set nexthop) you need to have a route to the provider's router (static or OSPF). From the sample config I guess OP may not realise that 'set nexthop' is on announcements, it doesn't overwrite the nexthop on incoming routes. Any confusion with the action of 'set' attributes, use bgpd -nv which demonstrates clearly how they apply. On 2006/10/12 14:05, X Y wrote: I use depend on carp0 ... carp3 on the master router (chosen via advskew) to drop that session if it fails, 'depend on carp' is mostly for where you have a peer session running from a carp address (e.g. if you're at an IXP where you are only allowed one IP address and want to connect two routers). It is used to hold the backup router in IDLE in normal conditions, and immediately connect when it becomes master. I'm not sure this is what you are intending to do? It might help to see some 'sh ip bgp d some.prefix'.
Re: Failover routers with OpenBGPD and independent BGP sessions
To me it seems that even having the IBGP session won't help the OP's particular issue (though he should have it anyway for other reasons)... as the peer session goes down, the routes from it go down with it, and IBGP withdraws those announcements. Nothing gets held over. Maybe establishing a second peering session with each provider will help in your redundancy- that's what I currently do, and yes, it has come in handy. I'm one week away from implementing OpenBGPD as route servers for maximum BGP redundancy. Here's my plan, maybe you can take something away from it for your own solution for redundancy- I have multiple egress points in my network (separate physical datacenters (DC's) with their own carriers, with the DC's linked together via Layer2 fiber access.) I use OpenBGPD as a route server (RS) at each DC to feed my edge routers a custom single table via IBGP (meaning the RS's are making the real EBGP multihop peerings with my carriers), making the edge routers essentially just packet forwarders. The DC's have layer2 access to each other, so I will populate each DC with an RS with CARP (in a VLAN) so the edge routers (and the carriers) only see one RS at a time, but multiple RS's are on standby. If a DC, edge router, RS, carrier, or Layer2 connections between the DC's fail, the RS's will be able to recover and continue to peer- either with everyone, or just their closest edge router and carriers (each DC has its own separate address blocks, so a split situation won't be a conflict.) As an extra step of redundancy against the RS's, I can have those edge routers form EBGP sessions with their directly connected peers, but only announcing a basic set of our prefixes (not individual /24's like the OpenBGPD RS's do for better traffic engineering, and only accepting a default route.) That way, if the entire redundant OpenBGPD router server model fails (which I doubt will happen), I still have basic routing with my carriers. Dan Farrell -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Stuart Henderson Sent: Friday, October 13, 2006 6:45 AM To: misc@openbsd.org Subject: Re: Failover routers with OpenBGPD and independent BGP sessions On 2006/10/13 11:24, Ronnie Garcia wrote: I have been recommended by our ISPs that I should also advertise routes between the routers, so that if one's BGP session fails, it can route packets to the other for a cleaner failover. I have not managed to get this configuration working. Yes you should, this is called iBGP. All of your BGP routers should have a iBGP session with all of the others, in a full mesh (unless you are using a route reflector). OP has already done that in the config file, the problem is how to add a route so the other provider's router can be reached. Normally the provider's router is listed in the IBGP announcement so unless this is overwritten in the IBGP announcements (by 'set nexthop) you need to have a route to the provider's router (static or OSPF). From the sample config I guess OP may not realise that 'set nexthop' is on announcements, it doesn't overwrite the nexthop on incoming routes. Any confusion with the action of 'set' attributes, use bgpd -nv which demonstrates clearly how they apply. On 2006/10/12 14:05, X Y wrote: I use depend on carp0 ... carp3 on the master router (chosen via advskew) to drop that session if it fails, 'depend on carp' is mostly for where you have a peer session running from a carp address (e.g. if you're at an IXP where you are only allowed one IP address and want to connect two routers). It is used to hold the backup router in IDLE in normal conditions, and immediately connect when it becomes master. I'm not sure this is what you are intending to do? It might help to see some 'sh ip bgp d some.prefix'.
Failover routers with OpenBGPD and independent BGP sessions
Hi,
I'm having a bit of trouble with the finer details of my OpenBGPD
config, and would appreciate some tips on getting it right and advice
on the right way of doing things.
I have two routers, two independent BGP connections, and a block of
provider independent address space. The routers are arranged in a
redundant pair. The public network and some private subnets have
gateway addresses provided with CARP. The two routers use pfsync.
The BGP connections are actually completely independent (I'll be
adding two more in due course for a total of four). They have
different network addresses, cables and route to the rest of the
world. The cables are plugged directly into the routers, and there's
no CARP on those interfaces. Packets will arrive via either of those
routes.
I have got a basic configuration working. This maintains the BGP
sessions, packets go in and out, and the firewalls will fail over as
they should. I use depend on carp0 ... carp3 on the master router
(chosen via advskew) to drop that session if it fails, and demote on
the backup to make sure it doesn't like being master if it doesn't
have a BGP session.
I have been recommended by our ISPs that I should also advertise
routes between the routers, so that if one's BGP session fails, it can
route packets to the other for a cleaner failover. I have not managed
to get this configuration working.
Some configuration information, with the real details removed to
protect the guilty.
AS: 9
PI subnet: A.A.A.0/23
PI gateway: A.A.A.1
Master: A.A.A.2
Backup: A.A.A.3
BGP connection 1: X.X.X.4 - X.X.X.200 on X.X.X.0/24, AS 8
BGP connection 2: Y.Y.Y.4 - Y.Y.Y.200 on Y.Y.Y.0/24, AS 8
(Y.Y.Y != X.X.X)
/etc/bgpd.conf
AS 9
network A.A.A.0/23
neighbor X.X.X.200 {
remote-as 8
local-address X.X.X.4
announce self
tcp md5sig password PASSWORD1
depend on carp1
depend on carp2
depend on carp3
# demote on backup
}
neighbor A.A.A.3 {
remote-as 9
descr backup
local-address A.A.A.2
announce all
tcp md5sig password PASSWORD2
set nexthop A.A.A.3# A.A.A.2 didn't help
set localpref -10
}
# ... filter rules as example.
# I have also tried adding
# deny from A.A.A.3 prefix A.A.A.0/23
I have checked the other is the mirror image. I have not included a
dmesg as the machine is working fine, and I have a basic configuration
working.
With the failover configuration, I have managed to get it to transfer
the routes across, but if I set the ISP session to idle, it appears to
reconfigure the routes (checked with bgpctl and route get), but the
packets never get through with no route to host. My test packets are
originated from the router with the idle BGP connection.
I am using pf, but the same effects are seen with it disabled.
Am I doing this right?
Is it necessary in the first place? It does appear to be a
belt-and-braces solution to the fail-over problem.
Many thanks.
Ben
