[
https://issues.apache.org/jira/browse/SOLR-8619?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15124761#comment-15124761
]
Shai Erera commented on SOLR-8619:
----------------------------------
There are two separate issues here:
* A new replica added via ADDREPLICA
* Which replica should become a leader
In the first case, I don't think that the replica should *ever* become the
leader, until it has had a chance to sync w/ a leader and first published its
state as ACTIVE. I also thought along the lines of Erick's suggestion, i.e. add
an INITIALIZING state to a Replica. Then, replicas can transition from
INITIALIZING -> RECOVERING -> ACTIVE, but never INITIALIZING -> DOWN. Then, the
DOWN -> ACTIVE transition is "safe" in that only non-initializing replicas can
become active leaders in the case of conflicts, or a whole cluster restart,
because we know they were once ACTIVE.
In case of a new collection, where all REPLICAS are new, then we have two
choices: either we note that in the internal ADDREPLICA call, so they are added
in DOWN state, or (which is simpler I guess), since all replicas will be
INITIALIZING, one can become the leader since they're all equal.
For the second case, which replica should become the leader, the proposals made
here make sense, but IMO they belong to a separate issue. Using the index
version, the commit point info etc. are good. But the problem that we've hit is
that the ONLY _live_ replica at the moment was the new (empty) one, there were
two others in DOWN state, but their nodes did not belong to the cluster (ZK
issues, network splits ...) and then that replica decided to become the leader.
When the 2 others later joined the cluster, they replicated "empty" index from
the leader, and data was lost.
If we added the new replica in the INITIALIZING state, it would stay that, and
when the two others returned to the cluster, they would re-compete for
leadership, using all the proposals made above, and no data would be lost.
> A new replica should not become leader when all current replicas are down as
> it leads to data loss
> --------------------------------------------------------------------------------------------------
>
> Key: SOLR-8619
> URL: https://issues.apache.org/jira/browse/SOLR-8619
> Project: Solr
> Issue Type: Bug
> Reporter: Anshum Gupta
>
> Here's what I'm talking about:
> * Start a 2 node solrcloud cluster
> * Create a 1 shard/1 replica collection
> * Add documents
> * Shut down the node that has the only active shard
> * ADDREPLICA for the shard/collection, so Solr would attempt to add a new
> replica on the other node
> * Solr waits for a while before this replica becomes an active leader.
> * Index a few new docs
> * Bring up the old node
> * The replica comes up, with it's old index and then syncs to only contain
> the docs from the new leader.
> All old documents are lost in this case
> Here are a few things that might work here:
> 1. Reject an ADDREPLICA call if all current replicas for the shard are down.
> Considering the new replica can not sync from anyone, it doesn't make sense
> for this replica to even come up
> 2. The replica shouldn't become active/leader unless either it was the last
> known leader or active before it went into recovering state
> unless there are no other replicas in the clusterstate.
> This might very well be related to SOLR-8173 but we should add a check to
> ADDREPLICA as well.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
