[
https://issues.apache.org/jira/browse/CASSANDRA-16518?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17366683#comment-17366683
]
Stefan Miklosovic edited comment on CASSANDRA-16518 at 6/21/21, 3:49 PM:
-------------------------------------------------------------------------
Few more observations to consider for [~JosephClay]
If you check what StorgeService#prepareToJoin does:
{code:java}
getTokenMetadata().updateHostId(localHostId,
FBUtilities.getBroadcastAddress());
appStates.put(ApplicationState.NET_VERSION,
valueFactory.networkVersion());
appStates.put(ApplicationState.HOST_ID,
valueFactory.hostId(localHostId));
appStates.put(ApplicationState.RPC_ADDRESS,
valueFactory.rpcaddress(FBUtilities.getBroadcastRpcAddress()));
appStates.put(ApplicationState.RELEASE_VERSION,
valueFactory.releaseVersion());
{code}
Release version is populated from FBUtilities.getReleaseVersionString():
{code:java}
public static String getReleaseVersionString()
{
try (InputStream in =
FBUtilities.class.getClassLoader().getResourceAsStream("org/apache/cassandra/config/version.properties"))
{
if (in == null)
{
return System.getProperty("cassandra.releaseVersion",
UNKNOWN_RELEASE_VERSION);
}
Properties props = new Properties();
props.load(in);
return props.getProperty("CassandraVersion");
}
catch (Exception e)
{
JVMStabilityInspector.inspectThrowable(e);
logger.warn("Unable to load version.properties", e);
return "debug version";
}
}
{code}
Are you sure that you have this file present? Also, are you sure that the
version string is indeed 3.11.6? It might be different if you are building your
own Cassandra distribution so it might have some unpredictable consequencies
when there is some abritrary version in it and you gossip it to other nodes and
so on. Does it strictly follow major.minor.patch format? If that resource is
not found, as you see, it will default to system property and if that one is
not set either, it will return "Unknown".
Then the Gossip is started:
{code:java}
Gossiper.instance.start(SystemKeyspace.incrementAndGetGeneration(), appStates);
// needed for node-ring gathering.
{code}
Check what incrementAndGetGeneration is doing, if that node is starting,
system.local is empty so it will generate its own generation but that
generation is ignored in case it is lower than the generations of other nodes
which might happen when time is "behind" on the joining node. Are you sure the
time is same on all nodes?
was (Author: stefan.miklosovic):
Few more observations to consider for [~JosephClay]
If you check what StorgeService#prepareToJoin does:
{code:java}
getTokenMetadata().updateHostId(localHostId,
FBUtilities.getBroadcastAddress());
appStates.put(ApplicationState.NET_VERSION,
valueFactory.networkVersion());
appStates.put(ApplicationState.HOST_ID,
valueFactory.hostId(localHostId));
appStates.put(ApplicationState.RPC_ADDRESS,
valueFactory.rpcaddress(FBUtilities.getBroadcastRpcAddress()));
appStates.put(ApplicationState.RELEASE_VERSION,
valueFactory.releaseVersion());
{code}
Release version is populated from FBUtilities.getReleaseVersionString():
{code:java}
public static String getReleaseVersionString()
{
try (InputStream in =
FBUtilities.class.getClassLoader().getResourceAsStream("org/apache/cassandra/config/version.properties"))
{
if (in == null)
{
return System.getProperty("cassandra.releaseVersion",
UNKNOWN_RELEASE_VERSION);
}
Properties props = new Properties();
props.load(in);
return props.getProperty("CassandraVersion");
}
catch (Exception e)
{
JVMStabilityInspector.inspectThrowable(e);
logger.warn("Unable to load version.properties", e);
return "debug version";
}
}
{code}
Are you sure that you have this file present? Also, are you sure that the
version string is indeed 3.11.6? It might be different if you are building your
own Cassandra distribution so it might have some unpredictable consequencies
when there is some abritrary version in it and you gossip it to other nodes and
so on. Does it strictly follow major.minor.patch format?
Then the Gossip is started:
{code:java}
Gossiper.instance.start(SystemKeyspace.incrementAndGetGeneration(), appStates);
// needed for node-ring gathering.
{code}
Check what incrementAndGetGeneration is doing, if that node is starting,
system.local is empty so it will generate its own generation but that
generation is ignored in case it is lower than the generations of other nodes
which might happen when time is "behind" on the joining node. Are you sure the
time is same on all nodes?
> Node restart during joining sets protocol version to V3
> -------------------------------------------------------
>
> Key: CASSANDRA-16518
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16518
> Project: Cassandra
> Issue Type: Bug
> Components: Messaging/Client
> Reporter: Joseph Clay
> Assignee: Stefan Miklosovic
> Priority: Normal
> Fix For: 3.11.x
>
>
> While joining nodes to a cluster, an old node crashed. The old node was
> recovered however clients (datastax java) refused to connect to it.
> The driver error:
> {noformat}
> Detected added or restarted Cassandra host /<ip>:<port> but ignoring it since
> it does not support the version V4 of the native protocol which is currently
> in use.{noformat}
> In the recovered node cassandra logs:
> {noformat}
> INFO o.a.c.transport.ConfiguredLimit Detected peers which do not fully
> support protocol V4. Capping max negotiable version to V3{noformat}
> I confirmed that ALL the nodes in the cluster, joining or otherwise, were
> apache-cassandra-3.11.6 so that error message was rather confusing.
> Eventually after digging through the code we got to the bottom of the issue:
> https://issues.apache.org/jira/browse/CASSANDRA-15193 adds a check for node
> version, which reverts the protocol version to V3 if any peer fails the
> version check. Joining nodes have NULL for their version in the peers table,
> which fails the version check.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
