[
https://issues.apache.org/jira/browse/CASSANDRA-16796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17384092#comment-17384092
]
Sam Tunnicliffe commented on CASSANDRA-16796:
---------------------------------------------
Thanks [~maedhroz]
bq. So just to make things explicit (for me, a non-gossip expert), notifying
subscribers in onChange() means we hit updateNormalTokens(), which removes the
endpoint from the "moving endpoints" set and eventually removes the pending
ranges?
Yes, that's right. The shutting down node is essentially returning to a
{{NORMAL}} state, so we just make sure that all
relevant parties (i.e. {{TokenMetadata}} ) are aware.
bq. What guarantees that a PendingRangeTask has run by the time
gossipShutdownUpdatesTokenMetadata() verifies there are no longer pending range
for node 2?
Good catch, I haven't seen any failures yet, but this definitely has the
potential for flakiness. I've added a log
statement at {{DEBUG}} at the end of {{Gossiper::markAsShutdown}} to gate the
test on, plus a blocking wait in the
assertion to ensure the PRT is complete before we check.
Circle runs are in the same place as before, new ASF CI jobs here:
[3.0|https://ci-cassandra.apache.org/view/patches/job/Cassandra-devbranch/961],
[4.0|https://ci-cassandra.apache.org/view/patches/job/Cassandra-devbranch/962]
> Clear pending ranges for a SHUTDOWN peer
> ----------------------------------------
>
> Key: CASSANDRA-16796
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16796
> Project: Cassandra
> Issue Type: Bug
> Components: Cluster/Membership
> Reporter: Sam Tunnicliffe
> Assignee: Sam Tunnicliffe
> Priority: Normal
> Fix For: 3.0.x, 3.11.x, 4.0.x
>
>
> If a node involved in a MOVE operation should fail, peers can sometimes
> maintain pending ranges for it even when it has left the ring and/or been
> replaced (in practice until the peer is next bounced). This in turn can lead
> to bogus unavailable responses to clients if a replica for the any of the
> pending ranges should go down.
> If the moving node crashes hard, a subsequent replacement will correctly fail
> as long as cassandra.consistent.rangemovement is set to true because the new
> node will learn the MOVING status from the remaining peers. A graceful
> shutdown, however, causes that status to be replaced with SHUTDOWN, but
> doesn't update TokenMetadata, so pending ranges remain for the down node,
> even after it has been removed from the ring.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
