Chris Miller created CASSANDRA-20910:
----------------------------------------
Summary: Instances from a 2nd ring join another ring when running
on the same nodes
Key: CASSANDRA-20910
URL: https://issues.apache.org/jira/browse/CASSANDRA-20910
Project: Apache Cassandra
Issue Type: Bug
Reporter: Chris Miller
Hi,
We experienced an issue today whereby instances from a 2nd ring join another
ring when running on the same nodes following a rolling restart which took
place following an OS patch and node reboot (both on Cassandra 4.1.2).
The cluster names and storage ports are different and this type of activity
normally runs without issue.
Any ideas as to what could have happened? Could this be a bug?
The seeds use the same IP addresses but no storage port is configured in the
seeds parameter, should we add the storage port to prevent this from happening
again? Any thoughts?
Messages like the following could be seen on ring 1.
INFO [GossipStage:1] 2025-09-18 04:11:49,040 Gossiper.java:1434 - Node
/XX.XX.XX.190:7002 is now part of the cluster
INFO [GossipStage:1] 2025-09-18 04:11:49,043 TokenMetadata.java:539 - Updating
topology for /XX.XX.XX.190:7002
INFO [Messaging-EventLoop-3-8] 2025-09-18 04:11:49,044
OutboundConnection.java:1153 -
/XX.XX.XX.61:7000(/XX.XX.XX.61:41920)->/XX.XX.XX.190:7002-URGENT_MESSAGES-7af53583
successfully connected, version = 12, framing = CRC, encryption = unencrypted
INFO [GossipStage:1] 2025-09-18 04:11:49,044 TokenMetadata.java:539 - Updating
topology for /XX.XX.XX.190:7002
INFO [GossipStage:1] 2025-09-18 04:11:49,044 Gossiper.java:1434 - Node
/XX.XX.XX.214:7002 is now part of the cluster
INFO [Messaging-EventLoop-3-3] 2025-09-18 04:11:49,046
OutboundConnection.java:1153 -
/XX.XX.XX.61:7000(/XX.XX.XX.61:62628)->/XX.XX.XX.214:7002-URGENT_MESSAGES-0515b24a
successfully connected, version = 12, framing = CRC, encryption = unencrypted
INFO [GossipStage:1] 2025-09-18 04:11:49,046 TokenMetadata.java:539 - Updating
topology for /XX.XX.XX.214:7002
INFO [GossipStage:1] 2025-09-18 04:11:49,046 TokenMetadata.java:539 - Updating
topology for /XX.XX.XX.214:7002
INFO [GossipStage:1] 2025-09-18 04:11:49,047 Gossiper.java:1434 - Node
/XX.XX.XX.247:7002 is now part of the cluster
INFO [Messaging-EventLoop-3-4] 2025-09-18 04:11:49,048
InboundConnectionInitiator.java:529 -
/XX.XX.XX.190:7002(/XX.XX.XX.190:60180)->/XX.XX.XX.61:7000-URGENT_MESSAGES-edfb2d8f
messaging connection established, version = 12, framing = LZ4, encryption =
unencrypted
Messages like the following in ring 2:
WARN [GossipStage:1] 2025-09-18 04:11:49,304
GossipDigestSynVerbHandler.java:58 - ClusterName mismatch from
/XX.XX.XX.247:7000 ring1!=ring2
WARN [GossipStage:1] 2025-09-18 04:11:49,819
GossipDigestSynVerbHandler.java:58 - ClusterName mismatch from
/XX.XX.XX.108:7000 ring1!=ring2
WARN [GossipStage:1] 2025-09-18 04:11:51,598
GossipDigestSynVerbHandler.java:58 - ClusterName mismatch from
/XX.XX.XX.190:7000 ring1!=ring2
WARN [GossipStage:1] 2025-09-18 04:11:52,361
GossipDigestSynVerbHandler.java:58 - ClusterName mismatch from
/XX.XX.XX.111:7000 ring1!=ring2
WARN [GossipStage:1] 2025-09-18 04:11:53,489
GossipDigestSynVerbHandler.java:58 - ClusterName mismatch from
/XX.XX.XX.84:7000 ring1!=ring2
WARN [GossipStage:1] 2025-09-18 04:11:58,322
GossipDigestSynVerbHandler.java:58 - ClusterName mismatch from
/XX.XX.XX.247:7000 ring1!=ring2
Instances from ring2 were listed in nodetool describecluster as unreachable
under schema versions.
They were also listed as DN under nodetool status.
The nodetool removenode command was used to remove the instances successfully.
Regards,
Chris.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
