Hi list, So googled, went through http://www.openbsd.org/faq/pf/carp.html a few times as well as the archives including one large thread which seemed to deal with this exact issue, but the solution was setting the VHID to the same on all carp interfaces (which I have already tried), and I can't see where I am screwing up.
CARP works, in terms of if I take one router down, the other router becomes master and when the first router comes back online, it preempts the master role back to itself. This is expected behaviour and works fine, I can reboot routers with impunity. What is not working, is if I stand on the master firewall, and "ifconfig carp0 down", then the carp0 goes into INIT, and the backup firewall carp0 goes into MASTER, however, the primary firewall carp1 still stays MASTER and the backup carp1 stays as BACKUP. As a consequence, traffic does not flow across the routers as you end up with: FW1 CARP0 - INIT FW1 CARP1 - MASTER FW2 CARP0 - MASTER FW2 CARP1 - BACKUP If I then "ifconfig carp1 down" on the master firewall I get: FW1 CARP0 - INIT FW1 CARP1 - INIT FW2 CARP0 - MASTER FW2 CARP1 - MASTER And traffic flows again. This seems contrary to http://www.openbsd.org/faq/pf/carp.html which states if you init one interface, then all carp interfaces on that redundancy group will advertise an infinite advskew. I have a pair of Soekris Net5501 routers with the following setup: +----| WAN/Internet |----+ | | |vr0| |vr0| +-----+ +-----+ | fw1 |-vr3----------vr3-| fw2 | +-----+ +-----+ | | |trunk1| |trunk1| | | ---+-------Shared LAN-------+--- Trunk1 on both routers are two NICs (vr1 & vr2) bonded in a trunk group Both routers are running 4.6 GENERIC#58 i386 On both firewalls, in pf.conf there is: # Top of pf.conf is: pfsync_if="vr3" carp_ext_if="carp0" carp_int_if="carp1" carpdevs="{ vr0 vr1 vr2 carp0_ext_if carp1_ext_if }" # .. skip tables, rdr, nat etc ... #near the top of the ruleset is: set skip on lo set skip on $pfsync_if pass quick on $carpdevs proto carp On both firewalls sysctl for carp is: $ sysctl | grep carp net.inet.carp.allow=1 net.inet.carp.preempt=1 net.inet.carp.log=2 FW1 hostname.if files are: $ cat /etc/hostname.carp0 inet 192.168.167.54 255.255.255.248 192.168.167.55 vhid 1 advskew 0 pass <password> $ cat /etc/hostname.carp1 inet 192.168.110.254 255.255.255.224 192.168.110.255 vhid 1 advskew 0 pass <password> $ cat /etc/hostname.pfsync0 up syncdev vr3 $ cat /etc/hostname.vr0 inet 192.168.167.52 255.255.255.248 NONE $ cat /etc/hostname.vr1 up $ cat /etc/hostname.vr2 up $ cat /etc/hostname.vr3 inet 172.16.0.1 255.255.255.252 NONE FW2 hostname.if files are: $ cat /etc/hostname.carp0 inet 192.168.167.54 255.255.255.248 192.168.167.55 vhid 1 advskew 128 pass <password> $ cat /etc/hostname.carp1 inet 192.168.110.254 255.255.255.224 192.168.110.255 vhid 1 advskew 128 pass <password> $ cat /etc/hostname.pfsync0 up syncdev vr3 $ cat /etc/hostname.vr0 inet 192.168.167.53 255.255.255.248 $ cat /etc/hostname.vr1 up $ cat /etc/hostname.vr2 up $ cat /etc/hostname.vr3 inet 172.16.0.2 255.255.255.252 NONE Netstat Returns: fw1 $ netstat -s -p carp carp: 34 packets received (IPv4) 0 packets received (IPv6) 0 packets discarded for bad interface 0 packets discarded for wrong TTL 0 packets shorter than header 0 discarded for bad checksums 0 discarded packets with a bad version 0 discarded because packet too short 0 discarded for bad authentication 0 discarded for unknown vhid 0 discarded because of a bad address list 580 packets sent (IPv4) 0 packets sent (IPv6) 0 send failed due to mbuf memory error 2 transitions to master fw1 $ netstat -s -p pfsync pfsync: 378 packets received (IPv4) 0 packets received (IPv6) 0 packets discarded for bad interface 0 packets discarded for bad ttl 0 packets shorter than header 0 packets discarded for bad version 0 packets discarded for bad HMAC 0 packets discarded for bad action 0 packets discarded for short packet 0 states discarded for bad values 0 stale states 290 failed state lookup/inserts 488 packets sent (IPv4) 0 packets sent (IPv6) 0 send failed due to mbuf memory error 0 send error fw2 $ netstat -s -p carp carp: 799 packets received (IPv4) 0 packets received (IPv6) 0 packets discarded for bad interface 0 packets discarded for wrong TTL 0 packets shorter than header 0 discarded for bad checksums 0 discarded packets with a bad version 0 discarded because packet too short 0 discarded for bad authentication 0 discarded for unknown vhid 0 discarded because of a bad address list 161 packets sent (IPv4) 0 packets sent (IPv6) 0 send failed due to mbuf memory error 3 transitions to master fw2 $ netstat -s -p pfsync pfsync: 869 packets received (IPv4) 0 packets received (IPv6) 0 packets discarded for bad interface 0 packets discarded for bad ttl 0 packets shorter than header 0 packets discarded for bad version 0 packets discarded for bad HMAC 0 packets discarded for bad action 0 packets discarded for short packet 0 states discarded for bad values 2 stale states 335 failed state lookup/inserts 1363 packets sent (IPv4) 0 packets sent (IPv6) 0 send failed due to mbuf memory error 0 send error ifconfig results on FW1 lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> mtu 33200 priority: 0 groups: lo inet 127.0.0.1 netmask 0xff000000 inet6 ::1 prefixlen 128 vr0: flags=8b43<UP,BROADCAST,RUNNING,PROMISC,ALLMULTI,SIMPLEX,MULTICAST> mtu 1500 lladdr 00:00:24:c9:a8:b8 priority: 0 groups: egress media: Ethernet autoselect (100baseTX full-duplex) status: active inet 192.168.167.52 netmask 0xfffffff8 broadcast 192.168.167.55 vr1: flags=8b43<UP,BROADCAST,RUNNING,PROMISC,ALLMULTI,SIMPLEX,MULTICAST> mtu 1500 lladdr 00:00:24:c9:a8:b9 priority: 0 trunk: trunkdev trunk1 media: Ethernet autoselect (100baseTX full-duplex) status: active vr2: flags=8b43<UP,BROADCAST,RUNNING,PROMISC,ALLMULTI,SIMPLEX,MULTICAST> mtu 1500 lladdr 00:00:24:c9:a8:b9 priority: 0 trunk: trunkdev trunk1 media: Ethernet autoselect (100baseTX full-duplex) status: active vr3: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500 lladdr 00:00:24:c9:a8:bb priority: 0 media: Ethernet autoselect (100baseTX full-duplex) status: active inet 172.16.0.1 netmask 0xfffffffc broadcast 172.16.0.3 enc0: flags=0<> mtu 1536 priority: 0 trunk1: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> mtu 1500 lladdr 00:00:24:c9:a8:b9 priority: 0 trunk: trunkproto failover trunkport vr2 trunkport vr1 master,active groups: trunk media: Ethernet autoselect status: active inet 192.168.110.251 netmask 0xffffffe0 broadcast 192.168.110.255 pfsync0: flags=41<UP,RUNNING> mtu 1500 priority: 0 pfsync: syncdev: vr3 maxupd: 128 defer: off groups: carp pfsync pflog0: flags=141<UP,RUNNING,PROMISC> mtu 33200 priority: 0 groups: pflog carp0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500 lladdr 00:00:5e:00:01:01 priority: 0 carp: MASTER carpdev vr0 vhid 1 advbase 1 advskew 0 groups: carp inet 192.168.167.54 netmask 0xfffffff8 broadcast 192.168.167.55 carp1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500 lladdr 00:00:5e:00:01:01 priority: 0 carp: MASTER carpdev trunk1 vhid 1 advbase 1 advskew 0 groups: carp inet 192.168.110.254 netmask 0xffffffe0 broadcast 192.168.110.255 ifconfig results on FW2 lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> mtu 33200 priority: 0 groups: lo inet 127.0.0.1 netmask 0xff000000 inet6 ::1 prefixlen 128 vr0: flags=8b43<UP,BROADCAST,RUNNING,PROMISC,ALLMULTI,SIMPLEX,MULTICAST> mtu 1500 lladdr 00:00:24:c9:a8:cc priority: 0 groups: egress media: Ethernet autoselect (100baseTX full-duplex) status: active inet 192.168.167.53 netmask 0xfffffff8 broadcast 192.168.167.55 vr1: flags=8b43<UP,BROADCAST,RUNNING,PROMISC,ALLMULTI,SIMPLEX,MULTICAST> mtu 1500 lladdr 00:00:24:c9:a8:cd priority: 0 trunk: trunkdev trunk1 media: Ethernet autoselect (100baseTX full-duplex) status: active vr2: flags=8b43<UP,BROADCAST,RUNNING,PROMISC,ALLMULTI,SIMPLEX,MULTICAST> mtu 1500 lladdr 00:00:24:c9:a8:cd priority: 0 trunk: trunkdev trunk1 media: Ethernet autoselect (100baseTX full-duplex) status: active vr3: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500 lladdr 00:00:24:c9:a8:cf priority: 0 media: Ethernet autoselect (100baseTX full-duplex) status: active inet 172.16.0.2 netmask 0xfffffffc broadcast 172.16.0.3 enc0: flags=0<> mtu 1536 priority: 0 trunk1: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> mtu 1500 lladdr 00:00:24:c9:a8:cd priority: 0 trunk: trunkproto failover trunkport vr2 trunkport vr1 master,active groups: trunk media: Ethernet autoselect status: active inet 192.168.110.252 netmask 0xffffffe0 broadcast 192.168.110.255 pfsync0: flags=41<UP,RUNNING> mtu 1500 priority: 0 pfsync: syncdev: vr3 maxupd: 128 defer: off groups: carp pfsync pflog0: flags=141<UP,RUNNING,PROMISC> mtu 33200 priority: 0 groups: pflog carp0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500 lladdr 00:00:5e:00:01:01 priority: 0 carp: BACKUP carpdev vr0 vhid 1 advbase 1 advskew 100 groups: carp inet 192.168.167.54 netmask 0xfffffff8 broadcast 192.168.167.55 carp1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500 lladdr 00:00:5e:00:01:01 priority: 0 carp: BACKUP carpdev trunk1 vhid 1 advbase 1 advskew 100 groups: carp inet 192.168.110.254 netmask 0xffffffe0 broadcast 192.168.110.255 Any help would be appreciated. Mikel
