[
https://issues.apache.org/jira/browse/CASSANDRA-13851?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16291566#comment-16291566
]
Sam Tunnicliffe commented on CASSANDRA-13851:
---------------------------------------------
This looks pretty reasonable, but the thing that concerns me is that the
'starting unsafely' path can be reached in 2 ways:
1) when a new node comes up and has no seeds (other than itself) configured,
and no persisted ring info.
2) when a non-bootstrapping node starts up with seeds and/or peers it *could*
contact, but it's unable to reach any of them within RING_DELAY.
I'm not worried about the former case, it is necessary for starting a single
node cluster (or the first node in a multi node cluster if starts are
staggered) and is also existing behaviour post CASSANDRA-10134.
The latter case though could lead to a regression of CASSANDRA-10134. If a new
node comes up with the same address as an existing down node, and it is not
bootstrapping (because it already bootstrapped, auto_bootstrap: false, or it's
present in its own seed list) it will take over that address if it cannot
perform the shadow round i.e. if partitioned.
bq. Imagine doing a full cluster bounce
There is already a mechanism in place to work around a whole cluster bounce. If
the whole cluster is restarted, all seeds would be in their own shadow rounds.
They report this to the any node contacting them and if all seeds are found to
be in that state, the node exits its own shadow round and continues to start.
In large clusters, you might have to increase RING_DELAY to give all seeds a
chance to enter their shadow round before other nodes start to time theirs out.
Extending the shadow round targets to include known peers works well for the
scenario where all seeds are down for a longer period, so I'm +1 on that. I
just think that to avoid regression of 10134 we still need to hard fail if a
node coming up is not able to contact any seeds *or* peers at all (unless case
1 above applies).
nits:
* 1 arg version of checkForEndpointCollision is unused
* Having 2 log statements at the end of doShadowRound is a bit meaningless, as
in both cases we proceed even though the state of the cluster is unknown.
> Allow existing nodes to use all peers in shadow round
> -----------------------------------------------------
>
> Key: CASSANDRA-13851
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13851
> Project: Cassandra
> Issue Type: Bug
> Components: Lifecycle
> Reporter: Kurt Greaves
> Assignee: Kurt Greaves
> Fix For: 3.11.x, 4.x
>
>
> In CASSANDRA-10134 we made collision checks necessary on every startup. A
> side-effect was introduced that then requires a nodes seeds to be contacted
> on every startup. Prior to this change an existing node could start up
> regardless whether it could contact a seed node or not (because
> checkForEndpointCollision() was only called for bootstrapping nodes).
> Now if a nodes seeds are removed/deleted/fail it will no longer be able to
> start up until live seeds are configured (or itself is made a seed), even
> though it already knows about the rest of the ring. This is inconvenient for
> operators and has the potential to cause some nasty surprises and increase
> downtime.
> One solution would be to use all a nodes existing peers as seeds in the
> shadow round. Not a Gossip guru though so not sure of implications.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
