[
https://issues.apache.org/jira/browse/CASSANDRA-11038?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15299397#comment-15299397
]
Cheng Ren commented on CASSANDRA-11038:
---------------------------------------
thanks everyone for fixing this issue!
> Is node being restarted treated as node joining?
> ------------------------------------------------
>
> Key: CASSANDRA-11038
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11038
> Project: Cassandra
> Issue Type: Bug
> Components: Distributed Metadata
> Reporter: cheng ren
> Assignee: Sam Tunnicliffe
> Priority: Minor
> Fix For: 2.2.x, 3.0.x, 3.x
>
>
> Hi,
> What we found recently is that every time we restart a node, all other nodes
> in the cluster treat the restarted node as a new node joining and issue node
> joining notification to clients. We have traced the code path being hit when
> a peer node detected a restarted node:
> src/java/org/apache/cassandra/gms/Gossiper.java
> {code}
> private void handleMajorStateChange(InetAddress ep, EndpointState epState)
> {
> if (!isDeadState(epState))
> {
> if (endpointStateMap.get(ep) != null)
> logger.info("Node {} has restarted, now UP", ep);
> else
> logger.info("Node {} is now part of the cluster", ep);
> }
> if (logger.isTraceEnabled())
> logger.trace("Adding endpoint state for " + ep);
> endpointStateMap.put(ep, epState);
> // the node restarted: it is up to the subscriber to take whatever
> action is necessary
> for (IEndpointStateChangeSubscriber subscriber : subscribers)
> subscriber.onRestart(ep, epState);
> if (!isDeadState(epState))
> markAlive(ep, epState);
> else
> {
> logger.debug("Not marking " + ep + " alive due to dead state");
> markDead(ep, epState);
> }
> for (IEndpointStateChangeSubscriber subscriber : subscribers)
> subscriber.onJoin(ep, epState);
> }
> {code}
> subscriber.onJoin(ep, epState) ends up with calling onJoinCluster in
> Server.java
> {code}
> src/java/org/apache/cassandra/transport/Server.java
> public void onJoinCluster(InetAddress endpoint)
> {
> server.connectionTracker.send(Event.TopologyChange.newNode(getRpcAddress(endpoint),
> server.socket.getPort()));
> }
> {code}
> We have a full trace of code path and skip some intermedia function calls
> here for being brief.
> Upon receiving the node joining notification, clients would go and scan
> system peer table to fetch the latest topology information. Since we have
> tens of thousands of client connections, scans from all of them put an
> enormous load to our cluster.
> Although in the newer version of driver, client skips fetching peer table if
> the new node has already existed in local metadata, we are still curious why
> node being restarted is handled as node joining on server side? Did we hit a
> bug or this is the way supposed to be? Our old java driver version is 1.0.4
> and cassandra version is 2.0.12.
> Thanks!
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
