I have a problem with significant packet loss rates whenever more than one machine using CARP is active. Here is some information on what I did so far:
I migrated an older (OpenBSD 3.8) firewall installation to OpenBSD 4.1 the other week and in the process upgraded it to use CARP, with two machines providing for failover security. While upgrading the firewall configuration worked out nicely after a bit of work, I was surprised to find that the moment I enabled CARP preemption on both machines last weekend, I suddenly had to put up with packet loss at a rate of between 5-25%. This I hadn't seen on the old firewall before. I checked that the packets were not simply getting dropped due to filtering rules, that the packet filter was not running out of memory, that there was no interrupt flooding, no significant IP packet corruption and no CPU load issues. The setup was OK, as far as I could tell. Except for the missing packets. Then I hit upon the idea to add a third machine to the CARP pool, with different Ethernet hardware, more memory, a different CPU, and OpenBSD 4.2, just to see if it made a difference. It did make a difference, since the packet loss rate increased (at some point, I was getting a packet loss rate of more than 80%). Eventually, I switched off two of the OpenBSD machines, leaving just one tending to the traffic. Strangely, the packet loss rate dropped to about 0% the moment it became master on both its CARP interfaces. All three machines would use the same CARP interface option of advskew=0 which I had assumed was safe to use, since the three would end up electing a master quickly enough. Apparently, I was wrong to make that assumption. My question is: is it generally unsafe to use CARP preemption without choosing different advskew values for each machine? Any advice would be appreciated. Thank you!
