[
https://issues.apache.org/jira/browse/CASSANDRA-20033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17893574#comment-17893574
]
David Capwell commented on CASSANDRA-20033:
-------------------------------------------
in 5.0 we changed the logic to no longer mutate using local versions and
instead the node shutting down sends its full state and the peers use that.
the previous logic had issues due to version values not being relative to the
node that went down, which lead to the cluster not converging properly...
in 5.0 if sees this message after the node started up again, then we would
override with the down state... but eventually should fix itself... we def
could do a check here to be extra safe.
> Shutdown message doesn't have generation check causing normal node considered
> shutdown by other nodes in cluster
> ----------------------------------------------------------------------------------------------------------------
>
> Key: CASSANDRA-20033
> URL: https://issues.apache.org/jira/browse/CASSANDRA-20033
> Project: Cassandra
> Issue Type: Bug
> Components: Cluster/Gossip
> Reporter: Runtian Liu
> Priority: Normal
> Fix For: 4.0.x, 4.1.x, 5.0.x
>
>
> Recently we run into one issue that when we do rolling restart of a cluster.
> We found one node is UN in it's own gossip state but all other nodes in the
> same cluster are considering it as a DN node.
> Gossip info from the node that is being considered as down by other nodes:
>
> {code:java}
> /1.1.1.1
> generation:1729812724
> heartbeat:20676
> STATUS:26:NORMAL,-1215648874011476782
> LOAD:20620:8.030878944E10
> SCHEMA:41:e3b3d9f6-e2e2-307b-959e-b493bd1f7bef
> DC:13:dc4
> RACK:15:dc4-0
> RELEASE_VERSION:6:4.1.3
> INTERNAL_IP:11:1.1.1.1
> RPC_ADDRESS:5:1.1.1.1
> NET_VERSION:2:12
> HOST_ID:3:65a87fef-e7f8-41f4-8cdf-9d8ea4f5e4f0
> RPC_READY:161:true
> INTERNAL_ADDRESS_AND_PORT:9:1.1.1.1:27378
> NATIVE_ADDRESS_AND_PORT:4:1.1.1.1:27379
> STATUS_WITH_PORT:25:NORMAL,-1215648874011476782
> SSTABLE_VERSIONS:7:big-nb
> TOKENS:24:<hidden> {code}
> Gossip state from other nodes for this node:
>
> {code:java}
> /1.1.1.1
> generation:1729812724
> heartbeat:2147483647
> STATUS:332020:shutdown,true
> LOAD:30:8.032911052E10
> SCHEMA:19:e3b3d9f6-e2e2-307b-959e-b493bd1f7bef
> DC:13:dc4
> RACK:15:dc4-0
> RELEASE_VERSION:6:4.1.3
> INTERNAL_IP:11:1.1.1.1
> RPC_ADDRESS:5:1.1.1.1
> NET_VERSION:2:12
> HOST_ID:3:65a87fef-e7f8-41f4-8cdf-9d8ea4f5e4f0
> RPC_READY:332021:false
> INTERNAL_ADDRESS_AND_PORT:9:1.1.1.1:27378
> NATIVE_ADDRESS_AND_PORT:4:1.1.1.1:27379
> STATUS_WITH_PORT:332020:shutdown,true
> SSTABLE_VERSIONS:7:big-nb
> TOKENS:24:<hidden> {code}
> The share the same generation but the other nodes are considering the node is
> shutdown.
>
>
> After closer look into the problem, I think here's what happened.
> When the node get restarted,
> 1. it first got gracefully shutdown, it will broadcast the GOSSIP_SHUTDOWN
> message to the rest of the cluster.
> 2. When it get back up, it will try to update it's generation and gossip with
> other nodes.
>
> If one node get the new generation for this 1.1.1.1 node first, then it
> receive the GOSSIP_SHUTDOWN message from step 1 (Assuming we have a very
> large delay in network between 1.1.1.1 node and the bad receiver node). We
> will run into above situation.
>
> I think GOSSIP_SHUTDOWN message should have the generation information and
> GossipShutdownVerbHandler should only bump the heartbeat for the local state
> if generation is same. If local generation is higher, we should ignore the
> shutdown message.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
