Re: pf/carp for redundant production use

[Neil]

Hi Jason, 

I would like to try your #1 suggestion but unfortunately, I don't know where 
to start. What are the programs I need? What configuration? Is there any 
existing sample configuration on a link that I can follow? 

Thanks for explaining this in very detail. 

Neil 

Jason Dixon writes: 

On Sep 25, 2005, at 8:30 AM, Neil wrote: 

Yep, the same behavior when the master dies. The solution that the  
person in #pf told me is use routing but I don't know how to  implement. 
He told me that it's an issue in pf's NAT.

Bullshit. 

Ok, here is the layman's description of the problem and the practical  
solution(s) to it.  I'd love to be able to explain why interfaces  
recovering from INIT don't reclaim MASTER faster than they do (approx  30 
seconds in my tests), but I don't understand the code-level  logistics of 
everything.  Hint:  This is only a problem using single  CARP hosts with 
preemption. 

PROBLEM: 

With a simple CARP design using a single CARP host on each segment  and 
preemption enabled, failover occurs as expected in the case of  any system 
offline condition (server crashes, admin reboots, etc).   If a single 
interface goes from MASTER to INIT state (cable gets  pulled, cable goes 
bad, card goes bad, etc), the 2nd interface on  that system will go into 
BACKUP mode as expected.  Traffic will route  across the new MASTER, and 
will continue to do so while the failed  system is in an INIT/BACKUP 
state. 

However, if the failed interface returns from INIT to an available  mode 
(we plug the cable in), we notice that the 2nd interface  reclaims MASTER 
almost immediately, but the restored interface does  not.  It becomes a 
BACKUP host, which leaves us with a routing  impossibility: 

BACKUP   MASTER
   carp0         carp0
      |                 |
   host1         host2
      |                 |
   carp1         carp1
MASTER   BACKUP 

Any internal clients will attempt to send traffic through the "new  
gateway" (host1), although neither system has any way of routing the  
traffic properly (not without some hokey static routes bypassing the  CARP 
hosts).  NOTE:  I have found that the original MASTER does  indeed return 
to the correct state, approximately 30 seconds later.   This is 
reproducible, but YMMV. 

SOLUTION: 

1) If you really are concerned about a partial system failure  (unplugged 
cable, bad card, etc), then scrap the single CARP host/ segment design and 
use arpbalance with multiple CARP hosts.  The same  partial-failure test 
using 2 CARP hosts on each segment with  arpbalance resulted in a perfect 
failover and recovery with no packet  loss. 

2) This is not tested, but I suspect that you should be able to use  the 
new interface grouping features in 3.8 to simply assign multiple  physical 
interfaces to the same group.  Even if one fails, the other  *should* 
maintain the MASTER state and avoid any partial failure  consequences.  
I'd love to hear from other users or developers that  have tried the 
grouping feature in this sort of scenario. 


--
Jason Dixon
DixonGroup Consulting
http://www.dixongroup.net