Hi friends,
I am trying to setup my first firewall w/failover via carp & pfsync. I
have it almost working, but am having a couple issues. I am hoping
someone will be able to help :)
First, before I enabled preemption I almost always had one machine being
master for one of the carp interfaces, and slave for the other two. It
seemed to work, but just looked troublesome. Enabling preemption seemed
to solve this. Does this point to a bigger problem somewhere?
Second, and what I am really trying to fix - is to have an in progress
TCP session fail over to the second firewall. The connection stalls and
eventually times out when failing over, but attempting to re-establish
after the failover works (through the second firewall). I've confirmed
(at least in my mind) that state updates are being properly propagated
to the second firewall by watching the pfsync interface, and noting the
state via pfctl -s state. I've watched syslog with pfctl -x loud and
didn't see anything.
Any hints on how I can go about troubleshooting this further? I've
included as much info as I can think of. The included PF ruleset is
just a proof of concept - I realize theres quite a bit more to be done,
I'm just trying to get the failover working.
Thanks!,
Tim
BTW If there is any OpenBSD guru in Calgary thats looking for a few
hours of consultancy I'd love to hear from you :)
Details:
Both systems are Dell 850 servers w/added Intel Etherexpress Pro 10/100
cards as the pfsync interface, with a crossover cable between them. OS
is OpenBSD 3.9, GENERIC Kernel.
192.168.1.246
+------------------+
| Test Workstation |
+------------------|
|
+----|---- carp1 ----|----+
| 192.168.1.22 |
| |
+----|---- carp2 ----|----+
| 192.168.1.23 |
| |
192.168.1.20 bge0| |bge0 192.168.1.21
+-----+ +-----+
| fw1 |-fxp0--------fxp0-| fw2 |
+-----+ +-----+
10.0.10.253 bge1| |bge1 10.0.10.254
| |
---+------- carp0 -------+---
10.0.10.1
|
|
+-------------+
| Test Server |
+-------------+
10.0.10.42
(fw1 fxp0 - 192.168.254.253)
(fs2 fxp0 - 192.168.254.254)
---- fw1:
# cat hostname.bge0
inet 192.168.1.20 255.255.255.0 NONE
# cat hostname.bge1
inet 10.0.10.253 255.255.255.0 NONE
# cat hostname.fxp0
inet 192.168.254.253 255.255.255.0 NONE
# cat hostname.carp0
inet 10.0.10.1 255.255.255.0 10.0.10.255 vhid 1 pass foo1 carpdev bge1
# cat hostname.carp1
inet 192.168.1.22 255.255.255.0 192.168.1.255 vhid 2 pass foo2 carpdev bge0
# cat hostname.carp2
inet 192.168.1.23 255.255.255.0 192.168.1.255 vhid 3 pass foo3 carpdev bge0
# cat hostname.pfsync0
up syncif fxp0
# sysctl -a | grep carp
net.inet.carp.allow=1
net.inet.carp.preempt=1
net.inet.carp.log=0
net.inet.carp.arpbalance=0
---- fw2:
# cat hostname.bge0
inet 192.168.1.21 255.255.255.0 NONE
# cat hostname.bge1
inet 10.0.10.254 255.255.255.0 NONE
# cat hostname.fxp0
inet 192.168.254.254 255.255.255.0 NONE
# cat hostname.carp0
inet 10.0.10.1 255.255.255.0 10.0.10.255 vhid 1 pass foo1 advskew 128
carpdev bge1
# cat hostname.carp1
inet 192.168.1.22 255.255.255.0 192.168.1.255 vhid 2 pass foo2 advskew
128 carpdev bge0
# cat hostname.carp2
192.168.1.23 255.255.255.0 192.168.1.255 vhid 3 pass foo3 advskew 128
carpdev bge0
# cat hostname.pfsync0
up syncif fxp0
# sysctl -a | grep carp
net.inet.carp.allow=1
net.inet.carp.preempt=1
net.inet.carp.log=0
net.inet.carp.arpbalance=0
---- PF Rules (identical on both machines)
# cat /etc/pf.conf
ext_if="bge0"
int_if="bge1"
pfsync_if="fxp0"
# All interfaces (real + virtual via carp) thought of as external
ext_ifs="{ bge0, carp1, carp2 }"
# Our internal network(s). Used for access rules and NAT
internal_nets="10.0.10.0/24"
# Define NAT source port range (all source ports will be rewritten to use
# this range)
nat_port_range="20001:65535"
# Define virtual carp interface that should be used as NAT source
# (i.e. outbound hide nat will appear to come from this virtual interface)
nat_carp="carp1"
# real interfaces that have virtual carp addresses associated with them
carp_interfaces="{ bge0, bge1 }"
# Test internal HTTP server
tstsrv_ext=192.168.1.22
tstsrv_int=10.0.10.42
tstsrv_port=80
###
### NAT
###
# Provide 'hide mode' nat for the entire subnet
nat on $ext_if from $internal_nets to any -> $nat_carp port $nat_port_range
# Test HTTP access
rdr on $ext_if proto tcp from any to $tstsrv_ext port $tstsrv_port ->
$tstsrv_int
###
### Access Rules
###
# Block and log everything by default
block log all
# Allow all localhost traffic
pass quick on lo0
# Allow pfsync traffic on pfsync interface
pass quick on $pfsync_if proto pfsync
# Allow carp traffic on all interfaces that have virtual carp addresses
# associated with them
pass quick on $carp_interfaces proto carp
# Allow communication to/from internal networks
pass in on $int_if from $internal_nets to any
pass out on $int_if from any to $internal_nets
# Allow firewall to communicate outbound
pass out on $ext_if from $ext_ifs to any keep state
# Allow HTTP test
pass in on $ext_if proto tcp from any to $tstsrv_int port $tstsrv_port
