Thanks Kanak for the explanation. It will definitely be very useful to have a few more knobs for tuning the rebalancing algorithm. I'll post a ticket soon.
On Oct 18, 2013, at 19:16 , Kanak Biscuitwala <[email protected]> wrote: > > Currently, the FULL_AUTO algorithm does not take this into account. The > algorithm optimizes for minimal movement and even distribution of states. > What I see here is that there is a tie in terms of even distribution, and > current presence of the replica would be a good tiebreaker. I can see why > this would be useful, though. Please create an issue and we'll pick it up > when we're able. > > On a somewhat related note, I noticed in your example code that you configure > and launch your nodes at the same time. The FULL_AUTO rebalancer performs > better when you configure your nodes ahead of time (even if you specify more > than you actually ever start). This is, of course, optional. Thanks for the advice. Currently we expect Helix to recompute states and partitions as nodes join the cluster, though indeed it's probably more efficient to compute some of the schedule ahead of time. I'll see how to apply your suggestion. Best regards, Matthieu > > Thanks, > Kanak > > From: Matthieu Morel <[email protected]> > Reply-To: "[email protected]" <[email protected]> > Date: Friday, October 18, 2013 10:03 AM > To: "[email protected]" <[email protected]> > Subject: Favoring some transitions when rebalancing in full_auto mode > > Hi, > > In FULL_AUTO mode, helix computes both partitioning and states. > > In a leader-replica model, I observed that when rebalancing due to a failure > of the Leader node, Helix does not promote an existing replica to leader, but > instead assigns a new leader (I.e. going from offline to replica to leader). > > For quick failover, we need to have the replica promoted to leader instead. > Is there a way to do so in FULL_AUTO mode? > > Apparently with SEMI_AUTO that would be possible, but it would imply we > control the partitioning, and we'd prefer Helix to control that as well. > > I tried to play with the priorities in the definition of the state model, > with no luck so far. > > (See the example below for an example of how rebalancing currently takes > place) > > Thanks! > > Matthieu > > > Here we have a deployment with 3 nodes, 3 partitions and 2 desired states, > Leader and Replica (and offline). > > // initial states > > "mapFields":{ > "MY_RESOURCE_0":{ > "instance_1":"REPLICA" > ,"instance_2":"LEADER" > } > ,"MY_RESOURCE_1":{ > "instance_0":"REPLICA" > ,"instance_1":"LEADER" > } > ,"MY_RESOURCE_2":{ > "instance_0":"LEADER" > ,"instance_2":"REPLICA" // Instance2 is replica > } > } > } > > > // instance 0 dies > > "mapFields":{ > "MY_RESOURCE_0":{ > "instance_1":"REPLICA" > ,"instance_2":"LEADER" > } > ,"MY_RESOURCE_1":{ > "instance_1":"LEADER" > ,"instance_2":"REPLICA" > } > ,"MY_RESOURCE_2":{ > "instance_1":"LEADER" // Helix preferred to assign leadership of > resource 2 to instance 1 rather than promoting instance_2 from replica to > leader > ,"instance_2":"REPLICA" // instance 2 is still replica for resource 2 > } > } > }
