[
https://issues.apache.org/jira/browse/SOLR-5495?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Timothy Potter updated SOLR-5495:
---------------------------------
Attachment: SOLR-5495.patch
Here's an updated patch that builds upon the previous one, same basic approach
of leader-initiated recovery but with some added coordination between the
leader and partitioned replica using a znode:
/collections/<collection>/leader_initiated_recovery/<shard>/<replicaCoreName>
(see ZkController.getLeaderInitiatedRecoveryZnodePath).
The basic idea here is in addition to the leader marking the replica down, a
separate znode is used to track the replica's transition from down ->
recovering -> active. So the leader marks the replica as down (which removes it
from participating in queries and update requests) and also creates this
special znode. When the replica finally gets the "you need to recover" command
from the leader, it changes the value of this znode to "recovering". When
recovery succeeds, the replica deletes the znode as it's no longer needed. If
the leader, while trying to send the recovery command (see
LeaderInitiatedRecoveryThread), sees the replica as being "active" but the
znode wasn't ack'd, then the leader can set the state to down again. As stated
before, I don't see where the replica would do this, but if it happens, we now
have a better way to handle it. Bottom line is with this special znode, the
replica cannot stay in the "active" state until it acks the leader's command to
recover by transitioning the znode appropriately.
The special znode is also useful if the nagging leader fails before the bad
replica receives the message. The idea here is that the new leader can query ZK
for any of these "leader-initiated-recovery" znodes for its shard and if there
are any in the "down" state, then it can start up the nag loop for each bad
replica; a znode in the down state means the replica hasn't received the
recovery command yet (see:
ElectionContext$ShardLeaderElectionContext.startLeaderInitiatedRecoveryOnReplicas).
There is a unit test that covers the leader failing over to a new leader and
resuming the "nag" loop on the downed replica. There's one area where I'm not
100% sure if it is correct yet ... in the shouldIBeLeader method in
ShardLeaderElectionContext, I check to see if a previous leader marked this
core "down" and if so, return false to indicate this node should not be the
leader. I think this works OK for RF=3 but I'm worried about RF=2 situations
where this check prevents a leader from being elected. The main idea behind
this check is that if the leader forces the shard state to "down", the
core.getCoreDescriptor().getCloudDescriptor().getLastPublished() method can
still return active so I needed this additional check on the znode. I suppose
we could try to update the lastPublished state when it changes but didn't see
how to go about that? (or if that was even a good idea).
Another area where I'm not 100% sold on is the 10-minute max wait and then
timeout loop in the LeaderInitiatedRecoveryThread. 10 mins is arbitrary but it
seems like it shouldn't just run forever. One idea I had was to use JMX to
raise some event / notification to allow monitoring tools to alert ops team of
this issue. Curious if there's anything else in SolrCloud related to notifying
of issues that need ops attention?
Lastly, I did give some thought to a self-initiating recovery approach where
the "trying to recover" loop runs on the replica itself as that is more immune
to leader changes and there's already a recover retry loop in place via the
RecoveryStrategy thread. As I understand it, a self-initiating approach would
work something like:
1) leader receives error when trying to forward update request to replica
2) leader marks replica as down in ZK
3) replica receives state change notification (at some point), replica must
iterate over all cores hosted on that node looking for cores marked as down
4) for each "down" core on the node found in step 3, try recovering in a loop
This is all straight-forward to implement. However, the main problem with this
approach is in step 4, when starting recovery, the replica updates its state to
"recovering" in ZK immediately. When a replica is "recovering" the leader still
tries to forward updates to it (the updates get stashed in the tlog until
recovery is complete). This works in normal circumstances because the replica
assumes there is no partition between it and the leader so it's OK to go into
the recovering state and continue receiving updates. The problem here though is
the network may still be partitioned, so the leader keeps trying to forward
docs and receiving errors. From the leader's perspective, we're right back at
step 1 above.
Of course, it would be possible to introduce a new state that would prevent the
leader from sending updates while the replica sorted itself out, but I'm
hesitant to introduce a new state as that has broader repercussions in the code
base. I'm mentioning this here in case someone else has some better ideas
around this self-initiating approach.
> Recovery strategy for leader partitioned from replica case.
> -----------------------------------------------------------
>
> Key: SOLR-5495
> URL: https://issues.apache.org/jira/browse/SOLR-5495
> Project: Solr
> Issue Type: Bug
> Reporter: Mark Miller
> Assignee: Timothy Potter
> Attachments: SOLR-5495.patch, SOLR-5495.patch
>
>
> We need to work out a strategy for the case of:
> Leader and replicas can still talk to ZooKeeper, Leader cannot talk to
> replica.
> We punted on this in initial design, but I'd like to get something in.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
