Re: isakmpd multiple tunnels
Thanks for the response. I should have been more clear. I am using isakmpd.conf and want to support multiple tunnels. Am I able to just add additional tunnels/lines under the [Phase 1] block that points to another relevant ISPEC configuration? Anyone? Thanks, Tim Hans-Joerg Hoexer wrote: On Thu, Apr 12, 2007 at 11:25:49AM -0600, Tim Pushor wrote: Hi friends, I'm looking to add another IPSEC connection to my openbsd 3.9 firewall. All examples I've seen are a single connection (phase 1). To support multiple vpn's tunnels, is it as simple as adding additional lines under [Phase 1] pointing to the new phase1 configuration block? yes. However, please take a look at ipsecctl(8) and ipsec.conf(5). HJ.
isakmpd multiple tunnels
Hi friends, I'm looking to add another IPSEC connection to my openbsd 3.9 firewall. All examples I've seen are a single connection (phase 1). To support multiple vpn's tunnels, is it as simple as adding additional lines under [Phase 1] pointing to the new phase1 configuration block? Thanks!
Re: Problem with MTU & IPSec VPN
Hi Darren, Just want to say thank you. You helped solve a problem thats been hounding me for a while now. It was in fact the smartdefence, but it was the number of fragmented packets allowed in a certain timespan. The problem was that I don't have access to the device and thus had to troubleshoot at one end of the connection until I was pretty sure it was the other side. Thats a crappy position to be in. Your help, coupled with the fact that I could see the pings arriving at the OpenBSD server and the response being sent out pointed to a problem on the other end. Now I just have to figure out why Path MTU discovery isn't working, but thats minor at this point. I'd love to send you a pizza of your choice. Please drop me an email and it'll be done. I'm serious. I'm SO relieved. Thanks, Tim Darren Spruell wrote: On 2/19/07, Tim Pushor <[EMAIL PROTECTED]> wrote: Hi all, I'm getting to the point where I don't really know where to turn. I am having a weird problem with an OpenBSD server/firewall that has a permament IPSec tunnel to a checkpoint embedded security device. The problem is, that half the time large packets can't get through. I've trial and error'ed (via windows ping -l) that ping packets 1306 bytes get through all the time, while packets > 1306 (even 1307) only get through half the time. Not half the time like 50% loss, but like it works for hours, then doesn't for 10 minutes. If that Check Point device has SmartDefense enabled, it has rules that futz with ICMP packets larger than some threshold. See if you have any of that mojo going on. DS
Problem with MTU & IPSec VPN
Hi all, I'm getting to the point where I don't really know where to turn. I am having a weird problem with an OpenBSD server/firewall that has a permament IPSec tunnel to a checkpoint embedded security device. The problem is, that half the time large packets can't get through. I've trial and error'ed (via windows ping -l) that ping packets 1306 bytes get through all the time, while packets > 1306 (even 1307) only get through half the time. Not half the time like 50% loss, but like it works for hours, then doesn't for 10 minutes. I only have control of one half of the connection (unfortunately), and am kind of lost. I have a rudimentary understanding of IP, so do understand things like MTU's & fragementation, and things like VPN adding packet overhead that reduces the effective MTU. I just don't really know where to start tracking this down. I guess I don't understand enough (any?) about how this part of the tunnel works under the covers. Can anyone help maybe point me in a direction? My ruleset is default deny with log, and nothing is being dropped. pfctl -x loud doesn't reveal anything. I pass everything to/from the VPN. My network configuration is unfortunately kind of complex. I have vlan's and carp's on those vlan's so I'll refrain from trying to describe the setup unless its necessary. Hopefully thanks in advance :) Tim
Re: watch traffic on IPSEC tunnel?
That was it, thank you :) Its been one of those days :) Jason Dixon wrote: On Feb 8, 2007, at 5:15 PM, Tim Pushor wrote: May be a dumb question, but how do I look at traffic going over an IPSEC tunnel, on one of the OpenBSD machines? I've tried tcpdump -i enc0 but get nothing .. The enc0 interface is down by default. Try bringing it up first ("ifconfig enc0 up"), then run your tcpdump.
watch traffic on IPSEC tunnel?
May be a dumb question, but how do I look at traffic going over an IPSEC tunnel, on one of the OpenBSD machines? I've tried tcpdump -i enc0 but get nothing ..
Strange vpn trouble
Hi friends, I am having a strange problem with a VPN that I've set up between an OpenBSD 3.9 server and a Checkpoint VPN-1 device. I've pretty much followed the guide at http://anubis.dweebsoft.com/HOWTO/isakmpd.html. I have to admit that I don't know enough about ipsec / isakmp. I do get some errors in the logfile: Feb 2 05:17:45 fw1 isakmpd[8492]: message_parse_payloads: invalid next payload type in payload of type 8 Feb 2 05:17:45 fw1 isakmpd[8492]: dropped message from 142.59.85.18 port 500 due to notification type INVALID_PAYLOAD_TYPE Feb 2 05:17:46 fw1 isakmpd[8492]: message_parse_payloads: invalid next payload type in payload of type 8 Feb 2 05:17:46 fw1 isakmpd[8492]: dropped message from 142.59.85.18 port 500 due to notification type INVALID_PAYLOAD_TYPE Feb 2 05:18:08 fw1 isakmpd[8492]: message_parse_payloads: reserved field non-zero: 1c Feb 2 05:18:08 fw1 isakmpd[8492]: dropped message from 142.59.85.18 port 500 due to notification type PAYLOAD_MALFORMED But the vpn seems to work. The weird problem I am having is that every so often, something strange happens and full packets don't seem to get through. Pings still get through, but when cranking up the packet size (with ping), once it its 1307, they stop. After an amount of time, things resume - and pings 1307+ get through again (and normal data starts flowing). This machine also routes between vlans and I havn't noticed any weirdness, although I am going to verify this. I'm really throwing this out because I don't know where to look. So far I've been focused on the key exchange but I'm starting to wonder if maybe its somewhere else. If anyone has a clue, I would REALLY appreciate it :) Thanks all, Tim
Re: Moving a 100GB directory tree with lots of hardlinks
Have you tried using cpio in passthrough mode? I've used CPIO on big systems before with success, although admittedly not on OpenBSD .. Matthias Bertschy wrote: OpenBSD 3.7 - i386 Pentium 4 3GHz - 1GB RAM - 2GB swap Hello list, For the past 3 weeks, I have been working on a difficult problem: moving a backuppc (http://backuppc.sourceforge.net/) pool from a RAID0 to a big disk, in order to free the RAID0 before rebuilding a RAID5. The RAID0 has one partition, its size is 2112984700 blocks (512-blocks), roughly 1008GB, which is close to the maximum allowed by ffs. The big disk is 300GB. I need to move 96GB of data which are, due to backuppc design, full of hardlinks! So far, I have tried to use: 1) dd: impossible because the partitions cannot be the same size (and the RAID5 won't be the same size as the RAID0) 2) pax -rw: after transferring almost 70GB, it bails out with a "Segmentation fault" 3) tar to archive: after something like 60GB, it complains with some "file name too long" errors 4) gtar to archive (from package gtar-1.15.1p0-static.tgz): ends up with a "gtar: memory exhauted" error 5) dump to file: successful but 5') restore from file: stops even before starting due to a "no memory for entry table" error (there is still a lot of unused memory and swap - and no ulimit) Any help is appreciated because I really don't know what to do next. Matthias Bertschy Echo Technologies SA
Re: Openbsd 3.9 + trunk
Steve Glaus wrote: Tim Pushor wrote: Steve Glaus wrote: Ok, I gotcha, trunk just looked like a ready mad solution for what I was trying to do... Could you tell me WHY it's not able to be used for that and what it is for? I've gone the pf route before to but it seems to add a lot of complexity to my ruleset trunk(4) is mainly used to provide redundancy or performance enhancement on the same network. I was using it to provide switch redundancy by putting one cable in one switch, one in the other, and the switches connected together. If I lose a switch, it keeps chugging along. Alright. Just so I understand.. COULD it be used to do what I'm trying to do? When you trunk two network interfaces together, are they adressless? Do the devices on the switch address the IP of the pseudo trunk interface? I don't know. I suspect you'd have routing issues, at least. Yes, other devices address the IP of the trunk interface, and my real interfaces are address-less.
Re: Openbsd 3.9 + trunk
Steve Glaus wrote: Ok, I gotcha, trunk just looked like a ready mad solution for what I was trying to do... Could you tell me WHY it's not able to be used for that and what it is for? I've gone the pf route before to but it seems to add a lot of complexity to my ruleset trunk(4) is mainly used to provide redundancy or performance enhancement on the same network. I was using it to provide switch redundancy by putting one cable in one switch, one in the other, and the switches connected together. If I lose a switch, it keeps chugging along.
Re: Website(s) being blocked by CARP/PF firewall
Again, does anyone have any ideas? Can other people access ticketmaster through their CARP'd NAT firewall? Yeah it works fine over here. How about cranking PF's debugging and watching syslog? pfctl -x loud Tim
Re: Hosting DNS from an openbsd cluster
Joachim Schipper wrote: It will work, but as noted, there's no particular reason to do this; redundancy is built into the DNS protocol. Well, there is a reason since I need another box to act as a secondary ;-) The only caveat I can think of is that running services on a firewall weakens your perimeter security. I concur. In this sealed environment it isn't nearly as much of a concern. The box is a router, with a very simple ruleset to allow remote administration over the Internet - thats the only real internet traffic. Finally, don't sync master and CARP - sync master and slave(s) directly. But that should be obvious. Yeah I thought that. I am still wondering if I should add the carp address for the secondary DNS (on the servers resolv.conf), or add secondary and tertiary addresses being the primary and backup router ... Regardless, I think you guys have answered my question. Thanks! Tim Joachim
Re: Hosting DNS from an openbsd cluster
Travers Buda wrote: Hi Friends, I am wondering anyone can think of why I shouldn't provide secondary DNS services from a carp cluster of OpenBSD systems? I have an issue where my primary DNS server is non-redundant, and trying to find a good place for a secondary. I have a cluster of OpenBSD machines acting as a router/firewall and would be real convenient to put it there. I'd like it to respond to queries on the carp address .. Can anyone think of a reason to not do this? Thanks, Tim You could use carp, but easier redundancy is already built into the DNS system. Look into a slave DNS server. Travers Buda Sorry, I should have been more clear. I am looking for a good spot on my network to put a secondary/slave DNS, and I already have a cluster of OpenBSD machines acting as a router/firewall and was wondering if there was any reason why not to use those as as slaves, since they are already redundant and highly available. Only question is to whether or not to use the/a carp address for the DNS. Thanks, Tim
Hosting DNS from an openbsd cluster
Hi Friends, I am wondering anyone can think of why I shouldn't provide secondary DNS services from a carp cluster of OpenBSD systems? I have an issue where my primary DNS server is non-redundant, and trying to find a good place for a secondary. I have a cluster of OpenBSD machines acting as a router/firewall and would be real convenient to put it there. I'd like it to respond to queries on the carp address .. Can anyone think of a reason to not do this? Thanks, Tim
Re: Redundant ethernet & Carp (was Re:Soekris)
Hi Joachim, Joachim Schipper wrote: On Thu, Aug 03, 2006 at 02:26:40PM -0600, Tim Pushor wrote: Well, after playing a little with trunk(4), etherchannel, and carp I am wondering something: Trying to achieve both firewall redundancy (via carp) and ethernet redundancy (via trunk(4)), would it be possible and (and maybe even recommended) to have firewall-1 connected solely to switch-1 and firewall-2 connected solely to switch-2, forgo the trunk(4), and just use carp to detect if either of the switches has failed, and fail over to the other switch/firewall combo? Am I making sense? I'm not entirely sure what you intend to achieve, but carp doesn't cross switches (it works on the local Ethernet segment). Really? I guess I don't understand enough about how carp works. I didn't see that as a limitation in any documentation that I read. Why exactly is this? Thanks, Tim
Redundant ethernet & Carp (was Re:Soekris)
Well, after playing a little with trunk(4), etherchannel, and carp I am wondering something: Trying to achieve both firewall redundancy (via carp) and ethernet redundancy (via trunk(4)), would it be possible and (and maybe even recommended) to have firewall-1 connected solely to switch-1 and firewall-2 connected solely to switch-2, forgo the trunk(4), and just use carp to detect if either of the switches has failed, and fail over to the other switch/firewall combo? Am I making sense? Thanks, Tim
Re: Soekris
Jason Dixon wrote: On Aug 1, 2006, at 5:23 PM, Tim Pushor wrote: Stuart Henderson wrote: The vlan idea makes a fair bit of sense - carp(4) over vlan(4) over trunk(4) over $some_nic(4) or some other mix - but if this is used for security be aware that your switch then becomes a security device. Google will find more information, including http://www.cisco.com/en/US/products/hw/switches/ps708/products_white_paper09186a008013159f.shtml Thinking about it more, isn't it carp over trunk over vlan over nic? I'm gonna give it a shot here in the next day or so. No. The OpenBSD trunk device is for link aggregation and failover, both properties of the physical layer. The OpenBSD vlan device provides access to VLAN segments (802.1Q), properties of the data-link and network layers. Again, make sure you're not confusing vendor terminologies here. An OpenBSD trunk is what many vendors refer to as teaming or bonding (some do refer to it as trunking). However, many(?) vendors refer to a trunk as a port carrying multiple tagged VLANs. Hi again Jason, Then I must have it wrong. I'll try to clarify: I have two boxes, with 3 interfaces total each. One interface goes to the other box for pfsync. That leaves 2 interfaces each. One interface will go to Ethernet switch 1, and one will go to Ethernet switch 2. Each interface will be split into 2 vlans, an internal and an external. Now I can team these vlan's together for redundancy. Perhaps I am thinking about this wrong .. Are you suggesting that I create a team of physical nics (using trunk(4)), then run vlans over that? Yes, I realize that trunk is used to refer to more than one thing. I have always thought of it though as teaming, but calling it trunk(4) as thats how it is documented in OpenBSD. And a huge thank you to all helping. Thanks, Tim
Re: Soekris
Stuart Henderson wrote: The vlan idea makes a fair bit of sense - carp(4) over vlan(4) over trunk(4) over $some_nic(4) or some other mix - but if this is used for security be aware that your switch then becomes a security device. Google will find more information, including http://www.cisco.com/en/US/products/hw/switches/ps708/products_white_paper09186a008013159f.shtml Thinking about it more, isn't it carp over trunk over vlan over nic? I'm gonna give it a shot here in the next day or so. I agree about the security issue. The alternative is to have 4 switches (for a redundant setup). Since we currently only have 4 (cheap) servers in the rack, its a tough sell to suggest that we need as many switches as servers, for simple Internet access. I am going to work on whether its even possible, then review that document (thanks for the link BTW) before making any final decisions. Thank you all for your help! Tim (I can post a summary with my conclusion if anyone is interested)
Re: Soekris
Hi Jason, Jason Dixon wrote: On Aug 1, 2006, at 3:13 PM, Tim Pushor wrote: Jason Dixon wrote: On Aug 1, 2006, at 2:48 PM, Tim Pushor wrote: Can anyone recommend another 4 port 10/100 ethernet card that will work well with OpenBSD 3.9? I don't have any recommendations on 4 port cards. If you have a switch that will support it, you should consider using VLANs with a gigabit card instead. Hmm now that is VERY interesting. Would it be possible to run a trunk on a vlan, then a carp on the trunk? Also, why the gigabit? Strictly performance? I think you're getting your technologies confused. If you're referring to an OpenBSD trunk (versus a Cisco trunk), that is an aggregation of physical ports on a switch. Theoretically, you would do this, then layer vlan interfaces on top of the trunk. However, you mentioned that you wanted 4 10/100 interfaces. Using a single gigabit port would enable you to exceed the capacity of 4 10/100 interfaces with a single port. You would have a single physical interface (say, em0) connected to a switch port enabled for VLANs (e.g., Cisco trunk). Then you can split up the networks by VLAN, rather than by physical connection. Here's a sample setup. Thanks a lot for replying. I am new to ethernet redudancy (and carp to boot) so I probably don't know what I'm talking about. We have a rack of servers that are now for the most part fully redundant. You can pull the plug on any box and nothing stops - almost. The ethernet switch is the last holdout. So now I am looking at adding ethernet redundancy to an already redundant firewall setup. So now instead of having 1 interface in, 1 interface out, and 1 interface pfsync, classically I'd need another 2 ports per server for redundancy. So I was thinking that instead of all this, I could run it all on vlans, if openbsd will do it. Am I wrong in thinking that I'd use a trunk(4) for a redundant ethernet connection? If not, then I was thinking that since the servers already have 2 gigabit ports on them (bge, from dell 850's) that I could run two vlan's each (one internal, one external), trunk(4) the vlans, then carp(4) the trunks.. Does that make sense? I'd still use the add-in card for pfsync. (I tried, but I suck at complex ascii art network diagrams) But to answer your question... no, it is not strictly a performance (higher throughput, fewer interrupts, etc) boost. Having less hardware means less opportunity for something to fail (ports, cables, etc). Gotcha. Thanks.. Tim
Re: Soekris
Jason Dixon wrote: On Aug 1, 2006, at 2:48 PM, Tim Pushor wrote: Can anyone recommend another 4 port 10/100 ethernet card that will work well with OpenBSD 3.9? I don't have any recommendations on 4 port cards. If you have a switch that will support it, you should consider using VLANs with a gigabit card instead. Hmm now that is VERY interesting. Would it be possible to run a trunk on a vlan, then a carp on the trunk? Also, why the gigabit? Strictly performance? Thanks, Tim
Soekris
Hi All, Not trying start a flame fest here (no, really). I am looking for multiport cards that work well with OpenBSD. Searching around the soekris cards seem to be a recommended solution. I seem to get sporadic and/or not very timely responses from soekris. I realize that they don't owe me anything and if they don't want to sell their products its their right. However, I am in need of a couple 4 port ethernet cards for use in a redundant firewall/ethernet solution and can't afford to wait to see if they will get back to me. (This attitude of mine isn't based on just one attempt at getting product from them) Can anyone recommend another 4 port 10/100 ethernet card that will work well with OpenBSD 3.9? Thanks! Tim
Re: Carp/Pfsync problem
Kian Mohageri wrote: On 7/31/06, Tim Pushor <[EMAIL PROTECTED]> wrote: Sorry to bump this thread, but I'd really like to know how to troubleshoot something like this. I'd suggest tcpdump'ing at the point when the connection fails, on the pflog(4) interface of both machines, especially the backup which is apparently dropping traffic after failover. Also, you haven't said whether there are any packet filters enabled on the client/server themselves, though I'd assume not. Thanks Kian - you are correct - they are just workstations on either side of the firewall cluster used for testing. They are wide open. I watched the log while attempting the failover. I block log all, so its the first place I look I also watched syslog running with pfctl -x loud, and verified that the state was properly propagated to the backup firewall. Anything else anyone can think of? Thanks, Tim
Re: Carp/Pfsync problem
Sorry to bump this thread, but I'd really like to know how to troubleshoot something like this. Should this work? Should I expect the firewall to fail over a TCP session? I'm thinking yes, since it does what its supposed to when shutting down the active firewall mid-stream, but not when I pull the plug on one. Thanks again, Tim Tim Pushor wrote: Hi friends, I am trying to setup my first firewall w/failover via carp & pfsync. I have it almost working, but am having a couple issues. I am hoping someone will be able to help :) First, before I enabled preemption I almost always had one machine being master for one of the carp interfaces, and slave for the other two. It seemed to work, but just looked troublesome. Enabling preemption seemed to solve this. Does this point to a bigger problem somewhere? Second, and what I am really trying to fix - is to have an in progress TCP session fail over to the second firewall. The connection stalls and eventually times out when failing over, but attempting to re-establish after the failover works (through the second firewall). I've confirmed (at least in my mind) that state updates are being properly propagated to the second firewall by watching the pfsync interface, and noting the state via pfctl -s state. I've watched syslog with pfctl -x loud and didn't see anything. Any hints on how I can go about troubleshooting this further? I've included as much info as I can think of. The included PF ruleset is just a proof of concept - I realize theres quite a bit more to be done, I'm just trying to get the failover working. Thanks!, Tim BTW If there is any OpenBSD guru in Calgary thats looking for a few hours of consultancy I'd love to hear from you :) Details: Both systems are Dell 850 servers w/added Intel Etherexpress Pro 10/100 cards as the pfsync interface, with a crossover cable between them. OS is OpenBSD 3.9, GENERIC Kernel. 192.168.1.246 +--+ | Test Workstation | +--| | +| carp1 |+ | 192.168.1.22 | | | +| carp2 |+ | 192.168.1.23 | || 192.168.1.20 bge0||bge0 192.168.1.21 +-+ +-+ | fw1 |-fxp0fxp0-| fw2 | +-+ +-+ 10.0.10.253 bge1||bge1 10.0.10.254 || ---+--- carp0 ---+--- 10.0.10.1 | | +-+ | Test Server | +-+ 10.0.10.42 (fw1 fxp0 - 192.168.254.253) (fs2 fxp0 - 192.168.254.254) fw1: # cat hostname.bge0 inet 192.168.1.20 255.255.255.0 NONE # cat hostname.bge1 inet 10.0.10.253 255.255.255.0 NONE # cat hostname.fxp0 inet 192.168.254.253 255.255.255.0 NONE # cat hostname.carp0 inet 10.0.10.1 255.255.255.0 10.0.10.255 vhid 1 pass foo1 carpdev bge1 # cat hostname.carp1 inet 192.168.1.22 255.255.255.0 192.168.1.255 vhid 2 pass foo2 carpdev bge0 # cat hostname.carp2 inet 192.168.1.23 255.255.255.0 192.168.1.255 vhid 3 pass foo3 carpdev bge0 # cat hostname.pfsync0 up syncif fxp0 # sysctl -a | grep carp net.inet.carp.allow=1 net.inet.carp.preempt=1 net.inet.carp.log=0 net.inet.carp.arpbalance=0 fw2: # cat hostname.bge0 inet 192.168.1.21 255.255.255.0 NONE # cat hostname.bge1 inet 10.0.10.254 255.255.255.0 NONE # cat hostname.fxp0 inet 192.168.254.254 255.255.255.0 NONE # cat hostname.carp0 inet 10.0.10.1 255.255.255.0 10.0.10.255 vhid 1 pass foo1 advskew 128 carpdev bge1 # cat hostname.carp1 inet 192.168.1.22 255.255.255.0 192.168.1.255 vhid 2 pass foo2 advskew 128 carpdev bge0 # cat hostname.carp2 192.168.1.23 255.255.255.0 192.168.1.255 vhid 3 pass foo3 advskew 128 carpdev bge0 # cat hostname.pfsync0 up syncif fxp0 # sysctl -a | grep carp net.inet.carp.allow=1 net.inet.carp.preempt=1 net.inet.carp.log=0 net.inet.carp.arpbalance=0 PF Rules (identical on both machines) # cat /etc/pf.conf ext_if="bge0" int_if="bge1" pfsync_if="fxp0" # All interfaces (real + virtual via carp) thought of as external ext_ifs="{ bge0, carp1, carp2 }" # Our internal network(s). Used for access rules and NAT internal_nets="10.0.10.0/24" # Define NAT source port range (all source ports will be rewritten to use # this range) nat_port_range="20001:65535" # Define virtual carp interface that should be used as NAT source # (i.e. outbound h
Re: Carp/Pfsync problem
Thanks Kian, That didn't have an effect. I suspected that it woudn't as I am getting state information, and ifconfig listed fxp0 as the syncdev even though I had syncif in the hostname file. As for the multiple carp addresses - This is in a lab environment but will end up protecting a rack of machines in a colo. I'm planning on having a carp address for each external address that's required (not many - maybe 4-5 eventually). Thanks, Tim Kian Mohageri wrote: Change 'syncif' to 'syncdev' in your hostname.pfsync files. Also, out of curiosity, why are there two CARP addresses between the workstation and firewalls? Kian On 9/20/06, Tim Pushor <[EMAIL PROTECTED]> wrote: Hi friends, I am trying to setup my first firewall w/failover via carp & pfsync. I have it almost working, but am having a couple issues. I am hoping someone will be able to help :) First, before I enabled preemption I almost always had one machine being master for one of the carp interfaces, and slave for the other two. It seemed to work, but just looked troublesome. Enabling preemption seemed to solve this. Does this point to a bigger problem somewhere? Second, and what I am really trying to fix - is to have an in progress TCP session fail over to the second firewall. The connection stalls and eventually times out when failing over, but attempting to re-establish after the failover works (through the second firewall). I've confirmed (at least in my mind) that state updates are being properly propagated to the second firewall by watching the pfsync interface, and noting the state via pfctl -s state. I've watched syslog with pfctl -x loud and didn't see anything. Any hints on how I can go about troubleshooting this further? I've included as much info as I can think of. The included PF ruleset is just a proof of concept - I realize theres quite a bit more to be done, I'm just trying to get the failover working. Thanks!, Tim BTW If there is any OpenBSD guru in Calgary thats looking for a few hours of consultancy I'd love to hear from you :) Details: Both systems are Dell 850 servers w/added Intel Etherexpress Pro 10/100 cards as the pfsync interface, with a crossover cable between them. OS is OpenBSD 3.9, GENERIC Kernel. 192.168.1.246 +--+ | Test Workstation | +--| | +| carp1 |+ | 192.168.1.22 | | | +| carp2 |+ | 192.168.1.23 | || 192.168.1.20 bge0||bge0 192.168.1.21 +-+ +-+ | fw1 |-fxp0fxp0-| fw2 | +-+ +-+ 10.0.10.253 bge1||bge1 10.0.10.254 || ---+--- carp0 ---+--- 10.0.10.1 | | +-+ | Test Server | +-+ 10.0.10.42 (fw1 fxp0 - 192.168.254.253) (fs2 fxp0 - 192.168.254.254) fw1: # cat hostname.bge0 inet 192.168.1.20 255.255.255.0 NONE # cat hostname.bge1 inet 10.0.10.253 255.255.255.0 NONE # cat hostname.fxp0 inet 192.168.254.253 255.255.255.0 NONE # cat hostname.carp0 inet 10.0.10.1 255.255.255.0 10.0.10.255 vhid 1 pass foo1 carpdev bge1 # cat hostname.carp1 inet 192.168.1.22 255.255.255.0 192.168.1.255 vhid 2 pass foo2 carpdev bge0 # cat hostname.carp2 inet 192.168.1.23 255.255.255.0 192.168.1.255 vhid 3 pass foo3 carpdev bge0 # cat hostname.pfsync0 up syncif fxp0 # sysctl -a | grep carp net.inet.carp.allow=1 net.inet.carp.preempt=1 net.inet.carp.log=0 net.inet.carp.arpbalance=0 fw2: # cat hostname.bge0 inet 192.168.1.21 255.255.255.0 NONE # cat hostname.bge1 inet 10.0.10.254 255.255.255.0 NONE # cat hostname.fxp0 inet 192.168.254.254 255.255.255.0 NONE # cat hostname.carp0 inet 10.0.10.1 255.255.255.0 10.0.10.255 vhid 1 pass foo1 advskew 128 carpdev bge1 # cat hostname.carp1 inet 192.168.1.22 255.255.255.0 192.168.1.255 vhid 2 pass foo2 advskew 128 carpdev bge0 # cat hostname.carp2 192.168.1.23 255.255.255.0 192.168.1.255 vhid 3 pass foo3 advskew 128 carpdev bge0 # cat hostname.pfsync0 up syncif fxp0 # sysctl -a | grep carp net.inet.carp.allow=1 net.inet.carp.preempt=1 net.inet.carp.log=0 net.inet.carp.arpbalance=0 PF Rules (identical on both machines) # cat /etc/pf.conf ext_if="bge0" int_if="bge1" pfsync_if="fxp0" # All interfaces (real + virtual via carp) thought of as ext
Carp/Pfsync problem
Hi friends, I am trying to setup my first firewall w/failover via carp & pfsync. I have it almost working, but am having a couple issues. I am hoping someone will be able to help :) First, before I enabled preemption I almost always had one machine being master for one of the carp interfaces, and slave for the other two. It seemed to work, but just looked troublesome. Enabling preemption seemed to solve this. Does this point to a bigger problem somewhere? Second, and what I am really trying to fix - is to have an in progress TCP session fail over to the second firewall. The connection stalls and eventually times out when failing over, but attempting to re-establish after the failover works (through the second firewall). I've confirmed (at least in my mind) that state updates are being properly propagated to the second firewall by watching the pfsync interface, and noting the state via pfctl -s state. I've watched syslog with pfctl -x loud and didn't see anything. Any hints on how I can go about troubleshooting this further? I've included as much info as I can think of. The included PF ruleset is just a proof of concept - I realize theres quite a bit more to be done, I'm just trying to get the failover working. Thanks!, Tim BTW If there is any OpenBSD guru in Calgary thats looking for a few hours of consultancy I'd love to hear from you :) Details: Both systems are Dell 850 servers w/added Intel Etherexpress Pro 10/100 cards as the pfsync interface, with a crossover cable between them. OS is OpenBSD 3.9, GENERIC Kernel. 192.168.1.246 +--+ | Test Workstation | +--| | +| carp1 |+ | 192.168.1.22 | | | +| carp2 |+ | 192.168.1.23 | || 192.168.1.20 bge0||bge0 192.168.1.21 +-+ +-+ | fw1 |-fxp0fxp0-| fw2 | +-+ +-+ 10.0.10.253 bge1||bge1 10.0.10.254 || ---+--- carp0 ---+--- 10.0.10.1 | | +-+ | Test Server | +-+ 10.0.10.42 (fw1 fxp0 - 192.168.254.253) (fs2 fxp0 - 192.168.254.254) fw1: # cat hostname.bge0 inet 192.168.1.20 255.255.255.0 NONE # cat hostname.bge1 inet 10.0.10.253 255.255.255.0 NONE # cat hostname.fxp0 inet 192.168.254.253 255.255.255.0 NONE # cat hostname.carp0 inet 10.0.10.1 255.255.255.0 10.0.10.255 vhid 1 pass foo1 carpdev bge1 # cat hostname.carp1 inet 192.168.1.22 255.255.255.0 192.168.1.255 vhid 2 pass foo2 carpdev bge0 # cat hostname.carp2 inet 192.168.1.23 255.255.255.0 192.168.1.255 vhid 3 pass foo3 carpdev bge0 # cat hostname.pfsync0 up syncif fxp0 # sysctl -a | grep carp net.inet.carp.allow=1 net.inet.carp.preempt=1 net.inet.carp.log=0 net.inet.carp.arpbalance=0 fw2: # cat hostname.bge0 inet 192.168.1.21 255.255.255.0 NONE # cat hostname.bge1 inet 10.0.10.254 255.255.255.0 NONE # cat hostname.fxp0 inet 192.168.254.254 255.255.255.0 NONE # cat hostname.carp0 inet 10.0.10.1 255.255.255.0 10.0.10.255 vhid 1 pass foo1 advskew 128 carpdev bge1 # cat hostname.carp1 inet 192.168.1.22 255.255.255.0 192.168.1.255 vhid 2 pass foo2 advskew 128 carpdev bge0 # cat hostname.carp2 192.168.1.23 255.255.255.0 192.168.1.255 vhid 3 pass foo3 advskew 128 carpdev bge0 # cat hostname.pfsync0 up syncif fxp0 # sysctl -a | grep carp net.inet.carp.allow=1 net.inet.carp.preempt=1 net.inet.carp.log=0 net.inet.carp.arpbalance=0 PF Rules (identical on both machines) # cat /etc/pf.conf ext_if="bge0" int_if="bge1" pfsync_if="fxp0" # All interfaces (real + virtual via carp) thought of as external ext_ifs="{ bge0, carp1, carp2 }" # Our internal network(s). Used for access rules and NAT internal_nets="10.0.10.0/24" # Define NAT source port range (all source ports will be rewritten to use # this range) nat_port_range="20001:65535" # Define virtual carp interface that should be used as NAT source # (i.e. outbound hide nat will appear to come from this virtual interface) nat_carp="carp1" # real interfaces that have virtual carp addresses associated with them carp_interfaces="{ bge0, bge1 }" # Test internal HTTP server tstsrv_ext=192.168.1.22 tstsrv_int=10.0.10.42 tstsrv_port=80 ### ### NAT ### # Provide 'hide mode' nat for the entire subnet nat on $ext_if from $internal_nets to any -> $nat_carp port $nat_port_range # Test HTTP access rdr on $ext_if proto tcp from any