Re: HA: pair of firewalls, 2 switches and 1 server
Am 21.05.2010 um 01:53 schrieb Tomoyuki Sakurai: You need additional two OSPF routers for L3 redundancy (claudio@ explained why in a paper). Thanks for the hint, Tomoyuki. I have now ospfd running on both firewalls, which was one necessary stop towards success. Axel --- axel@chaos1.de PGP-Key:29E99DD6 +49 151 2300 9283 computing @ chaos claudius
Re: HA: pair of firewalls, 2 switches and 1 server
Am 21.05.2010 um 12:55 schrieb Axel Rau: Am 20.05.2010 um 22:07 schrieb Reyk Floeter: I will try the following with unmanaged switches, no RST: On fbsd: fbsd# ifconfig em0 up fbsd# ifconfig em1 up fbsd# ifconfig lagg0 create fbsd# ifconfig lagg0 laggproto failover laggport em0 laggport em1 up fbsd# ifconfig vlan2 create fbsd# ifconfig vlan2 vlan 2 vlandev lagg0 10.1.2.10 netmask 255.255.255.0 up This started working with 2 unmanaged switches after applying a patch to fbsd.8.0 (bug with vlan on top of lagg). Thanks again Reyk for your help, Axel --- axel@chaos1.de PGP-Key:29E99DD6 +49 151 2300 9283 computing @ chaos claudius
Re: HA: pair of firewalls, 2 switches and 1 server
On Fri, May 21, 2010 at 12:22:10AM +0200, r...@openbsd.org wrote: Linux's bonding module has an arp monitor which solves some of these problems, but the implementation is so hackish (as usual there...) that I'd rather not use it in production. arping and ifstated might do the same on openbsd, but I'm not sure if that will work when the interfaces are trunk ports. I'll need to check this when I have time. why not? trunk is just a normal ethernet interface. the linux bondage trick sounds hackish, but link detection protocols like udld or bfd should help here on the ethernet level. many managed switches support one of these protocols and i'd like to do this on the openbsd side at some point to alter the link state based on optional uni-/bidirectional link detection. If one of the 2 ports of the switch where you are connected is badly configured (wrong VLAN, ...), you may have problems. The link can be up but being unusable. In that case, having something like the arp monitor seems to not be so stupid; doesn't it ? -- Olivier Cherrier - Symacx.com mailto:o...@symacx.com
Re: HA: pair of firewalls, 2 switches and 1 server
On Fri, May 21, 2010 at 12:22:10AM +0200, Reyk Floeter wrote: Linux's bonding module has an arp monitor which solves some of these problems, but the implementation is so hackish (as usual there...) that I'd rather not use it in production. arping and ifstated might do the same on openbsd, but I'm not sure if that will work when the interfaces are trunk ports. I'll need to check this when I have time. why not? trunk is just a normal ethernet interface. the monitoring should be done on the ports/slaves/child interfaces, not the trunk itself. I don't see why arping wouldn't work on those, either, but I haven't tested it. the linux bondage trick sounds hackish, but link detection protocols like udld or bfd should help here on the ethernet level. many managed switches support one of these protocols and i'd like to do this on the openbsd side at some point to alter the link state based on optional uni-/bidirectional link detection. This would be a pretty good out of the box solution. end to end monitoring with ifstated would still be useful especially on the end hosts, which can just (ar)ping the carp gateway and kick out interfaces that can't reach it. That would work against config mistakes (missing vlans) and all kinds of subtle switch failures. For the routers this is not so easy, they would need to ping an assortment of end hosts to get a really useful end to end check. And there is always relayd et al that solve the problem even better (in the cases where it can be used.)
Re: HA: pair of firewalls, 2 switches and 1 server
Am 20.05.2010 um 22:07 schrieb Reyk Floeter: I will try the following with unmanaged switches, no RST: +---+ +--+ |fw1|+-+ | | +em1++ sw1 +---+ | carp0|em2+--+ +-+-+-+em0| | | | | | | | | +-+-+ | ++ | | | || | | |Server| +-+-+ +--|+ | | fbsd | |fw2| || | | | |em1+-+ +-+-+-+ | | +em2++ sw2 +---+ | carp0| |+-+em1| | +---+ +--+ vlan1+vlan2 vlan2 fw1# ifconfig em0 up fw1# ifconfig em1 up fw1# ifconfig trunk0 trunkport em0 trunkport em1 trunkproto failover up fw1# ifconfig vlan1 vlandev trunk0 descr UPLINK 10.1.1.2/24 fw1# ifconfig vlan2 vlandev trunk0 descr SERVERLAN 10.1.2.2/24 fw1# ifconfig carp1 vhid 1 carpdev vlan1 10.1.1.1/24 fw1# ifconfig carp2 vhid 2 carpdev vlan2 10.1.2.1/24 fw2# ifconfig em0 up fw2# ifconfig em1 up fw2# ifconfig trunk0 trunkport em0 trunkport em1 trunkproto failover up fw2# ifconfig vlan1 vlandev trunk0 descr UPLINK 10.1.1.3/24 fw2# ifconfig vlan2 vlandev trunk0 descr SERVERLAN 10.1.2.3/24 fw2# ifconfig carp1 vhid 1 carpdev vlan1 advskew 100 10.1.1.1/24 fw2# ifconfig carp2 vhid 2 carpdev vlan2 advskew 100 10.1.2.1/24 On fbsd: fbsd# ifconfig em0 up fbsd# ifconfig em1 up fbsd# ifconfig lagg0 create fbsd# ifconfig lagg0 laggproto failover laggport em0 laggport em1 up fbsd# ifconfig vlan2 create fbsd# ifconfig vlan2 vlan 2 vlandev lagg0 10.1.2.10 netmask 255.255.255.0 up fbsd# route add default 10.1.2.1 Axel --- axel@chaos1.de PGP-Key:29E99DD6 +49 151 2300 9283 computing @ chaos claudius
Re: HA: pair of firewalls, 2 switches and 1 server
Am 20.05.2010 um 00:04 schrieb Henning Brauer: * Axel Rau axel@chaos1.de [2010-05-19 10:34]: Now the question: Can I put a trunk on top of a carp? you put carp on top of the trunk of course. OK. Can I have a trunk connected to 2 different switches then? Axel --- axel@chaos1.de PGP-Key:29E99DD6 +49 151 2300 9283 computing @ chaos claudius
Re: HA: pair of firewalls, 2 switches and 1 server
On Thu, May 20, 2010 at 07:02:23PM +0200, Axel Rau wrote: Am 20.05.2010 um 00:04 schrieb Henning Brauer: * Axel Rau axel@chaos1.de [2010-05-19 10:34]: Now the question: Can I put a trunk on top of a carp? you put carp on top of the trunk of course. OK. Can I have a trunk connected to 2 different switches then? Not normally. Some higher-end switches can support this, eg the HP Procurve switches running their K-series software can do something they call distributed trunking (and no doubt Cisco and other vendors all call it something else). But as I think you were talking about using cheapish Netgear switches it's unlikely to be possible. -- - Graham Allan School of Physics and Astronomy - University of Minnesota -
Re: HA: pair of firewalls, 2 switches and 1 server
* Graham Allan al...@physics.umn.edu [2010-05-20 19:23]: On Thu, May 20, 2010 at 07:02:23PM +0200, Axel Rau wrote: Am 20.05.2010 um 00:04 schrieb Henning Brauer: * Axel Rau axel@chaos1.de [2010-05-19 10:34]: Now the question: Can I put a trunk on top of a carp? you put carp on top of the trunk of course. OK. Can I have a trunk connected to 2 different switches then? Not normally. Some higher-end switches can support this, eg the HP Procurve switches running their K-series software can do something they call distributed trunking (and no doubt Cisco and other vendors all call it something else). But as I think you were talking about using cheapish Netgear switches it's unlikely to be possible. well, lacp usually doesn't work across switches. but lacp is not the only mode trunk supports. roundrobin definately works across switches - how well might depend on your switches. works well for me on procurve with E-series software which doesn't do distributed trunking afair. -- Henning Brauer, h...@bsws.de, henn...@openbsd.org BS Web Services, http://bsws.de Full-Service ISP - Secure Hosting, Mail and DNS Services Dedicated Servers, Rootservers, Application Hosting
Re: HA: pair of firewalls, 2 switches and 1 server
On Thu, May 20, 2010 at 07:28:55PM +0200, Henning Brauer wrote: * Graham Allan al...@physics.umn.edu [2010-05-20 19:23]: On Thu, May 20, 2010 at 07:02:23PM +0200, Axel Rau wrote: Am 20.05.2010 um 00:04 schrieb Henning Brauer: * Axel Rau axel@chaos1.de [2010-05-19 10:34]: Now the question: Can I put a trunk on top of a carp? you put carp on top of the trunk of course. OK. Can I have a trunk connected to 2 different switches then? Not normally. Some higher-end switches can support this, eg the HP Procurve switches running their K-series software can do something they call distributed trunking (and no doubt Cisco and other vendors all call it something else). But as I think you were talking about using cheapish Netgear switches it's unlikely to be possible. well, lacp usually doesn't work across switches. but lacp is not the only mode trunk supports. roundrobin definately works across switches - how well might depend on your switches. works well for me on procurve with E-series software which doesn't do distributed trunking afair. How about the warnings about packet reordering and interactions with TCP? I'd guess it's not really such a big issue if you have two identical switches and routers. But shouldn't the hash based trunk modes work just fine, too (with the caveat that some flows will stop working completely if the other switch fails in some ways while roundrobin will cause half of the packets to be blackholed, keeping badly degraded connectivity) Also, the switches need to be separate; connecting them directly may cause learned MACs to flap between the real host port and the cable between the switches and make the trunk receive its own traffic on the other port. Fail-over trunk should work just fine, too. But see the following paragraphs... If you want reliability, do not use cheap switches. Switch power supplies are not the failure mode you want to avoid. I don't remember seeing very many at all, however I've seen lots of crappy ones lose their config or stop forwarding completely while keeping the link up. I have two identical core switches in one (not really so critical at all) place running OSPF, with a bunch of routers connecting to both switches for redundancy. Works pretty well and there has even been a config reset incident, which didn't break anything - because OSPF can detect link failures. Trying to do the same all the way to the end hosts (i.e. without a routing protocol) is pretty difficult. One pseudo solution is to run a bridge instead of trunk on the 2 interfaces and use STP for fail-over; I find that too yucky to solve a problem that doesn't really exist (just buy a reliable switch with a redundant power supply or connect the single one to a good UPS) However, if you need to ask if you can run a trunk on top of a carp, do yourself a favor and use a single switch. There will be less downtime. Jussi Peltola
Re: HA: pair of firewalls, 2 switches and 1 server
* Jussi Peltola pe...@pelzi.net [2010-05-20 20:07]: On Thu, May 20, 2010 at 07:28:55PM +0200, Henning Brauer wrote: * Graham Allan al...@physics.umn.edu [2010-05-20 19:23]: On Thu, May 20, 2010 at 07:02:23PM +0200, Axel Rau wrote: Am 20.05.2010 um 00:04 schrieb Henning Brauer: * Axel Rau axel@chaos1.de [2010-05-19 10:34]: Now the question: Can I put a trunk on top of a carp? you put carp on top of the trunk of course. OK. Can I have a trunk connected to 2 different switches then? Not normally. Some higher-end switches can support this, eg the HP Procurve switches running their K-series software can do something they call distributed trunking (and no doubt Cisco and other vendors all call it something else). But as I think you were talking about using cheapish Netgear switches it's unlikely to be possible. well, lacp usually doesn't work across switches. but lacp is not the only mode trunk supports. roundrobin definately works across switches - how well might depend on your switches. works well for me on procurve with E-series software which doesn't do distributed trunking afair. How about the warnings about packet reordering and interactions with TCP? never ran into such issues. too lazy right now to check wether trunk deals with that in roundrobin or wether i just got lucky. I'd guess it's not really such a big issue if you have two identical switches and routers. But shouldn't the hash based trunk modes work just fine, too (with the caveat that some flows will stop working completely if the other switch fails in some ways while roundrobin will cause half of the packets to be blackholed, keeping badly degraded connectivity) err. wait. if the switch fails for real the link goes down and the port is just taken out of the active ports on the trunk. now there are of course more subtle ways of failure that could lead to the above scenario. but how likely is that really? and would this issue be your real problem then? Also, the switches need to be separate; connecting them directly may cause learned MACs to flap between the real host port and the cable between the switches and make the trunk receive its own traffic on the other port. that is the may depend on your switch part. I have not seen any problems with interconnected procurves, 5300XL series. Fail-over trunk should work just fine, too. indeed. If you want reliability, do not use cheap switches. Switch power supplies are not the failure mode you want to avoid. I don't remember seeing very many at all, however I've seen lots of crappy ones lose their config or stop forwarding completely while keeping the link up. guess i lack the cheap shit switch experience. i do have experience with expensive shit switches tho. they suck in many different ways, never seen the behaviour you describe above tho. but then, ever since using said procurves, that is history. I have two identical core switches in one (not really so critical at all) place running OSPF, with a bunch of routers connecting to both switches for redundancy. Works pretty well and there has even been a config reset incident, which didn't break anything - because OSPF can detect link failures. Trying to do the same all the way to the end hosts (i.e. without a routing protocol) is pretty difficult. i would never ever run any L3 on switches. However, if you need to ask if you can run a trunk on top of a carp, do yourself a favor and use a single switch. There will be less downtime. that is something i could subscribe to :) -- Henning Brauer, h...@bsws.de, henn...@openbsd.org BS Web Services, http://bsws.de Full-Service ISP - Secure Hosting, Mail and DNS Services Dedicated Servers, Rootservers, Application Hosting
Re: HA: pair of firewalls, 2 switches and 1 server
* Henning Brauer lists-open...@bsws.de [2010-05-20 20:23]: * Jussi Peltola pe...@pelzi.net [2010-05-20 20:07]: On Thu, May 20, 2010 at 07:28:55PM +0200, Henning Brauer wrote: * Graham Allan al...@physics.umn.edu [2010-05-20 19:23]: On Thu, May 20, 2010 at 07:02:23PM +0200, Axel Rau wrote: Am 20.05.2010 um 00:04 schrieb Henning Brauer: * Axel Rau axel@chaos1.de [2010-05-19 10:34]: Now the question: Can I put a trunk on top of a carp? you put carp on top of the trunk of course. OK. Can I have a trunk connected to 2 different switches then? Not normally. Some higher-end switches can support this, eg the HP Procurve switches running their K-series software can do something they call distributed trunking (and no doubt Cisco and other vendors all call it something else). But as I think you were talking about using cheapish Netgear switches it's unlikely to be possible. well, lacp usually doesn't work across switches. but lacp is not the only mode trunk supports. roundrobin definately works across switches - how well might depend on your switches. works well for me on procurve with E-series software which doesn't do distributed trunking afair. How about the warnings about packet reordering and interactions with TCP? never ran into such issues. too lazy right now to check wether trunk deals with that in roundrobin or wether i just got lucky. uh, I just checked and... I am actually running failover. oups. -- Henning Brauer, h...@bsws.de, henn...@openbsd.org BS Web Services, http://bsws.de Full-Service ISP - Secure Hosting, Mail and DNS Services Dedicated Servers, Rootservers, Application Hosting
Re: HA: pair of firewalls, 2 switches and 1 server
On Thu, May 20, 2010 at 08:17:48PM +0200, Henning Brauer wrote: I have two identical core switches in one (not really so critical at all) place running OSPF, with a bunch of routers connecting to both switches for redundancy. Works pretty well and there has even been a config reset incident, which didn't break anything - because OSPF can detect link failures. Trying to do the same all the way to the end hosts (i.e. without a routing protocol) is pretty difficult. i would never ever run any L3 on switches. Bad wording on my part, the routers run OSPF and the switches are dumb L2 devices. Still, without OSPF et al there would be no way to detect a crappy switch failing in funny ways, which was my point. As an extra note, if you do get a crappy switch, be very careful with its management interface. The cheapest ones have unbelievably slow CPUs that are easily overloaded by broadcasts making the whole thing stop responding. Even worse, the interrupt load seems to trigger some other bugs, like LACP mysteriously failing and disabling one port on a trunk and blackholing half of your traffic (this happened on a ZyXEL GS-4024, which has otherwise totally Just Worked as a L2 switch for years) or even the whole switch ASIC crashing after a broadcast storm and requiring a reboot (though the management CPU was still responding through the out of band ether and serial port after the storm was gone) Also, it's a very obvious DoS; a malicious person needs to send a rather small amount of BPDUs to overload the tiny CPU and the cheap switches obviously have no rate limiting for packets going to the CPU (only on all broadcasts). So, blocking BPDUs from non-trusted devices should be enabled (but that should probably be done anyway.) Even among trusted devices STP and LACP involve the shitty code running on the underpowered management CPU, and that is not the part that shines in the cheap switches. Static link aggregation works OK.
Re: HA: pair of firewalls, 2 switches and 1 server
On Thu, May 20, 2010 at 08:17:48PM +0200, Henning Brauer wrote: * Jussi Peltola pe...@pelzi.net [2010-05-20 20:07]: If you want reliability, do not use cheap switches. Switch power supplies are not the failure mode you want to avoid. I don't remember seeing very many at all, however I've seen lots of crappy ones lose their config or stop forwarding completely while keeping the link up. guess i lack the cheap shit switch experience. i do have experience with expensive shit switches tho. they suck in many different ways, never seen the behaviour you describe above tho. but then, ever since using said procurves, that is history. I agree with the don't use cheap switches statement. If you look at the price of (eg) used procurve 2824's then I don't see why anyone would use Netgear or suchlike. It's also good to have a switch with a real management interface that can help you tell what's going on. Graham
Re: HA: pair of firewalls, 2 switches and 1 server
Am 20.05.2010 um 20:17 schrieb Henning Brauer: However, if you need to ask if you can run a trunk on top of a carp, This was an academic question to keep the thread running (-; do yourself a favor and use a single switch. There will be less downtime. that is something i could subscribe to :) I try to keep things simple usually. Thanks to all for the advice. Axel --- axel@chaos1.de PGP-Key:29E99DD6 +49 151 2300 9283 computing @ chaos claudius
Re: HA: pair of firewalls, 2 switches and 1 server
On Thu, May 20, 2010 at 07:02:23PM +0200, Axel Rau wrote: Now the question: Can I put a trunk on top of a carp? you put carp on top of the trunk of course. OK. Can I have a trunk connected to 2 different switches then? yes, i did this many times using trunk in failover mode. this is actually the main reason why i implemented failover mode: for l2 redundancy. i even normally use it in combination with VLANs. to explain it using your artwork: +---+ +--+ | |+-+ | | +fw1++ sw1 +---+ | carp0| +--+ +-+-+-+em0| | | | | | | | +-+-+ | ++ | | || ||Server| +-+-+ +--|--+ | fbsd | | | | | | | | +-+ +-+-+-+ | | +fw2++ sw2 +---+ | carp0| |+-+em1| | +---+ +--+ let's assume that fw1 and fw2 are connected with em1 and em2, em1 is connected to sw1 and em2 is connected to sw2 on each fw. fbsd server sits in vlan2, the uplink is in vlan1 connected to the same switches (you might also have other physical switches for the uplink, which is also fairly common, which would just require to move vlan1 to another trunk or physical iface). the switches don't need any special configuration, no trunks on the switch and no stacking or similar. they just need to be in the same VLANs, so a simple interlink between them is all you need. failover mode means that the trunk only uses one active link at a time (the first trunkport you add and so on) as long as the link is up. this is works nicely with any kind of switches, is safe to use and doesn't cause any loops, address conflicts etc.. i use procurve switches (now: hp networking e-series), but there is no need for distributed trunking or tricks like this with failover mode. fw1# ifconfig em0 up fw1# ifconfig em1 up fw1# ifconfig trunk0 trunkport em0 trunkport em1 trunkproto failover up fw1# ifconfig vlan1 vlandev trunk0 descr UPLINK 10.1.1.2/24 fw1# ifconfig vlan2 vlandev trunk0 descr SERVERLAN 10.1.2.2/24 fw1# ifconfig carp1 vhid 1 carpdev vlan1 10.1.1.1/24 fw1# ifconfig carp2 vhid 2 carpdev vlan2 10.1.2.1/24 fw2# ifconfig em0 up fw2# ifconfig em1 up fw2# ifconfig trunk0 trunkport em0 trunkport em1 trunkproto failover up fw2# ifconfig vlan1 vlandev trunk0 descr UPLINK 10.1.1.3/24 fw2# ifconfig vlan2 vlandev trunk0 descr SERVERLAN 10.1.2.3/24 fw2# ifconfig carp1 vhid 1 carpdev vlan1 advskew 100 10.1.1.1/24 fw2# ifconfig carp2 vhid 2 carpdev vlan2 advskew 100 10.1.2.1/24 and you can also move the pfsync traffic over the same trunk: fw1# ifconfig vlan240 vlandev trunk0 192.168.240.2/24 up fw1# ifconfig pfsync0 syncdev vlan240 up fw2# ifconfig vlan240 vlandev trunk0 192.168.240.3/24 up fw2# ifconfig pfsync0 syncdev vlan240 up reyk
Re: HA: pair of firewalls, 2 switches and 1 server
I do this too. In addition to the previously mentioned problems with cheap switches losing their configs (and vlans) you should make sure the active interfaces are all on one switch so that the link between them isn't uselessly used; this will also avoid an unpleasant split brain event if that link ever happens to fail. But in this case you will also have to very carefully check the other switch stays properly configured so the backup interfaces will actually pass the traffic you want. Linux's bonding module has an arp monitor which solves some of these problems, but the implementation is so hackish (as usual there...) that I'd rather not use it in production. arping and ifstated might do the same on openbsd, but I'm not sure if that will work when the interfaces are trunk ports. I'll need to check this when I have time.
Re: HA: pair of firewalls, 2 switches and 1 server
Thanks for this detailed elaboration, Reyk. A few questions: Am 20.05.2010 um 22:07 schrieb Reyk Floeter: On Thu, May 20, 2010 at 07:02:23PM +0200, Axel Rau wrote: Now the question: Can I put a trunk on top of a carp? you put carp on top of the trunk of course. OK. Can I have a trunk connected to 2 different switches then? yes, i did this many times using trunk in failover mode. this is actually the main reason why i implemented failover mode: for l2 redundancy. i even normally use it in combination with VLANs. to explain it using your artwork: +---+ +--+ | |+-+ | | +fw1++ sw1 +---+ | carp0| +--+ +-+-+-+em0| | | | | | | | +-+-+ | ++ | | || ||Server| +-+-+ +--|--+ | fbsd | | | | | | | | +-+ +-+-+-+ | | +fw2++ sw2 +---+ | carp0| |+-+em1| | +---+ +--+ let's assume that fw1 and fw2 are connected with em1 and em2, em1 is connected to sw1 and em2 is connected to sw2 on each fw. fbsd server sits in vlan2, the uplink is in vlan1 connected to the same switches (you might also have other physical switches for the uplink, which is also fairly common, which would just require to move vlan1 to another trunk or physical iface). the switches don't need any special configuration, no trunks on the switch and no stacking or similar. they just need to be in the same VLANs, so a simple interlink between them is all you need. You mean a physical connection between sw1 and sw2? failover mode means that the trunk only uses one active link at a time (the first trunkport you add and so on) as long as the link is up. this is works nicely with any kind of switches, is safe to use and doesn't cause any loops, address conflicts etc.. i use procurve switches (now: hp networking e-series), but there is no need for distributed trunking or tricks like this with failover mode. fw1# ifconfig em0 up fw1# ifconfig em1 up fw1# ifconfig trunk0 trunkport em0 trunkport em1 trunkproto failover up fw1# ifconfig vlan1 vlandev trunk0 descr UPLINK 10.1.1.2/24 fw1# ifconfig vlan2 vlandev trunk0 descr SERVERLAN 10.1.2.2/24 fw1# ifconfig carp1 vhid 1 carpdev vlan1 10.1.1.1/24 fw1# ifconfig carp2 vhid 2 carpdev vlan2 10.1.2.1/24 fw2# ifconfig em0 up fw2# ifconfig em1 up fw2# ifconfig trunk0 trunkport em0 trunkport em1 trunkproto failover up fw2# ifconfig vlan1 vlandev trunk0 descr UPLINK 10.1.1.3/24 fw2# ifconfig vlan2 vlandev trunk0 descr SERVERLAN 10.1.2.3/24 fw2# ifconfig carp1 vhid 1 carpdev vlan1 advskew 100 10.1.1.1/24 fw2# ifconfig carp2 vhid 2 carpdev vlan2 advskew 100 10.1.2.1/24 On fbsd, I set default gw to 10.1.1.1 ? But a trunk would have no counter parts. How does this fit in? fbsd# ifconfig em0 up fbsd# ifconfig em1 up fbsd# ifconfig lagg0 laggproto failover laggport em0 laggport em1 \ 10.1.2.10 netmask 255.255.255.0 ? Axel --- axel@chaos1.de PGP-Key:29E99DD6 +49 151 2300 9283 computing @ chaos claudius
Re: HA: pair of firewalls, 2 switches and 1 server
On Thu, May 20, 2010 at 11:31:22PM +0300, Jussi Peltola wrote: I do this too. In addition to the previously mentioned problems with cheap switches losing their configs (and vlans) you should make sure the active interfaces are all on one switch so that the link between them isn't uselessly used; this will also avoid an unpleasant split brain event if that link ever happens to fail. But in this case you will also have to very carefully check the other switch stays properly configured so the backup interfaces will actually pass the traffic you want. don't mix up cheap switches with crap switches. actually, some very expensive switches are really crappy indeed. but i don't see your problems, you just have to take care a little bit and don't try to run your highly redundant high-performance firewall cluster with a bunch of SOHO linksys switches (oh wait, they're cisco now). but there is no real problem, trunk failover with carp + pfsync and redundant switches works very well and i have installed it in many different highly available production sites. it is hard to make it not work unless you configure your switches wrong - eg. by cascading the redundant switches to other uplink switches and creating some weird loops. Linux's bonding module has an arp monitor which solves some of these problems, but the implementation is so hackish (as usual there...) that I'd rather not use it in production. arping and ifstated might do the same on openbsd, but I'm not sure if that will work when the interfaces are trunk ports. I'll need to check this when I have time. why not? trunk is just a normal ethernet interface. the linux bondage trick sounds hackish, but link detection protocols like udld or bfd should help here on the ethernet level. many managed switches support one of these protocols and i'd like to do this on the openbsd side at some point to alter the link state based on optional uni-/bidirectional link detection. reyk
Re: HA: pair of firewalls, 2 switches and 1 server
On Tue, May 18, 2010 at 10:32 PM, Axel Rau axel@chaos1.de wrote: Yes, but what carps/trunks do I need? I'm doing carp(4)+pfsync(4)+bridge(4)+vether(4)+trunk(4)+ospfd(8) for L3/L2 redundancy. Part of my config can be found at: http://cvs.openbsd.org/cgi-bin/query-pr-wrapper?full=yesnumbers=6318 You need additional two OSPF routers for L3 redundancy (claudio@ explained why in a paper). -- Tomoyuki Sakurai
Re: HA: pair of firewalls, 2 switches and 1 server
Axel Rau schrieb: Am 18.05.2010 um 14:11 schrieb Guido Tschakert: I would say your Server is __the__ single point of failure (sure the switch is also a spof but normally I'm more worried about servers then switches) Yes, but it has 2 power supplies and redundant disks. If the mini pwr supply of the single switch dies, I'm loosing. Oh, yes and it has two mainboards and you have two ups for all the things. ;-) Have you thought of two internet connections from two different providers? Sorry I don't want to bother you, I just want to say that achieving redundancy is not that easy like you described it in your first message. The first question is: What problem are you trying to resolve? Or in your case: How much redundancy do you want/need? Also a 2nd server is in the pipeline... Ok, that's fine. guido Btw: it would be great for the archive, if you got it working, that you send a message to the list, describing your configuration.
Re: HA: pair of firewalls, 2 switches and 1 server
Am 19.05.2010 um 07:59 schrieb Guido Tschakert: What problem are you trying to resolve? I will clarify: +---+ +--+ | |+-+ | | +fw1++ sw1 +---+ | carp0| +--+ +-+-+-+em0| | | | | | | | +-+-+ | ++ | | || ||Server| +-+-+ +--|--+ | fbsd | | | | | | | | +-+ +-+-+-+ | | +fw2++ sw2 +---+ | carp0| |+-+em1| | +---+ +--+ Server uses fw1/fw2 as default gateway(s). Server has a bunch of IPs. I can't add these as aliases to either em0 or em1 (would be single point o failure). I need a virtual interface, like a trunk, to which I can tie the IPs. A trunk connects 2 hosts (AFAIK), in my case, I have 3. I could reduce the pair fw1/fw2 to one virtual system, using 2 carp interfaces. This way, I would have a valid configuration of 2 hosts for the trunk, with 2 interfaces on each side. Now the question: Can I put a trunk on top of a carp? AFAIK No. What do you mean? Are there other possibilities to connect the boxes with the above functionality? Axel --- axel@chaos1.de PGP-Key:29E99DD6 +49 151 2300 9283 computing @ chaos claudius
Re: HA: pair of firewalls, 2 switches and 1 server
* Axel Rau axel@chaos1.de [2010-05-19 10:34]: Now the question: Can I put a trunk on top of a carp? you put carp on top of the trunk of course. -- Henning Brauer, h...@bsws.de, henn...@openbsd.org BS Web Services, http://bsws.de Full-Service ISP - Secure Hosting, Mail and DNS Services Dedicated Servers, Rootservers, Application Hosting
Re: HA: pair of firewalls, 2 switches and 1 server
Axel Rau wrote: Hi all, I have a pair of redundant firewalls (obsd 4.6) and a server (fbsd 8.0): +---+ +--+ | | | | +fw1+--+ +-+ | carp0| |carp1 | | em0| | | | | | | | +-+-++-+-+-+ | | | | sw | |Server| +-+-++-+-+-+ | fbsd | | | | | | | +fw2+--+ +-+ | carp0| |carp1 em1| | | | | | +---+ DMZ +--+ We all know, the switch is the sigle point of failure. Even worse, when it fails the carp0 pair starts flapping, disturbing other firewall traffic. So, how to resolve this? Trunking would only be possible between 2 boxes, not 3. Carp on top of trunk? 2 Carp pairs on the firewalls and 1 pair at the server? If I get it right, the physical LAN should look like this: +---+ +--+ | |+-+ | | +fw1++ sw1 +---+ | carp0| +--+ +-+-+-+em0| | | | | | | | +-+-+ | ++ | | || ||Server| +-+-+ +--|--+ | fbsd | | | | | | | | +-+ +-+-+-+ | | +fw2++ sw2 +---+ | carp0| |+-+em1| | +---+ +--+ Switches must have Spanning Tree support (RSTP), so I hope a pair of Netgear GS108T can do this. Any proposals highly appreciated, Axel --- axel@chaos1.de PGP-Key:29E99DD6 +49 151 2300 9283 computing @ chaos claudius IMHO, the second scenario you draw solves the problem in a very elegant way. Beside, STP and RSTP-enabled switches are becoming less expansive in the last years. Best regards.
Re: HA: pair of firewalls, 2 switches and 1 server
Am 18.05.2010 um 14:20 schrieb Leonardo Carneiro - Veltrac: IMHO, the second scenario you draw solves the problem in a very elegant way. Beside, STP and RSTP-enabled switches are becoming less expansive in the last years. Yes, but what carps/trunks do I need? Axel --- axel@chaos1.de PGP-Key:29E99DD6 +49 151 2300 9283 computing @ chaos claudius
Re: HA: pair of firewalls, 2 switches and 1 server
Axel Rau schrieb: Hi all, I have a pair of redundant firewalls (obsd 4.6) and a server (fbsd 8.0): +---+ +--+ | | | | +fw1+--+ +-+ | carp0| |carp1 | | em0| | | | | | | | +-+-++-+-+-+ | | | | sw | |Server| +-+-++-+-+-+ | fbsd | | | | | | | +fw2+--+ +-+ | carp0| |carp1 em1| | | | | | +---+ DMZ +--+ We all know, the switch is the sigle point of failure. Hi, I would say your Server is __the__ single point of failure (sure the switch is also a spof but normally I'm more worried about servers then switches) guido Even worse, when it fails the carp0 pair starts flapping, disturbing other firewall traffic. So, how to resolve this? Trunking would only be possible between 2 boxes, not 3. Carp on top of trunk? 2 Carp pairs on the firewalls and 1 pair at the server? If I get it right, the physical LAN should look like this: +---+ +--+ | |+-+ | | +fw1++ sw1 +---+ | carp0| +--+ +-+-+-+em0| | | | | | | | +-+-+ | ++ | | || ||Server| +-+-+ +--|--+ | fbsd | | | | | | | | +-+ +-+-+-+ | | +fw2++ sw2 +---+ | carp0| |+-+em1| | +---+ +--+ Switches must have Spanning Tree support (RSTP), so I hope a pair of Netgear GS108T can do this. Any proposals highly appreciated, Axel --- axel@chaos1.de PGP-Key:29E99DD6 +49 151 2300 9283 computing @ chaos claudius
Re: HA: pair of firewalls, 2 switches and 1 server
Am 18.05.2010 um 14:11 schrieb Guido Tschakert: I would say your Server is __the__ single point of failure (sure the switch is also a spof but normally I'm more worried about servers then switches) Yes, but it has 2 power supplies and redundant disks. If the mini pwr supply of the single switch dies, I'm loosing. Also a 2nd server is in the pipeline... Axel --- axel@chaos1.de PGP-Key:29E99DD6 +49 151 2300 9283 computing @ chaos claudius
