I have a 9 node cluster I had shut down (cassandra stopped on all nodes, all nodes shutdown) that I just tried to start back up. I have done this several times successfully. However, on this attempt, one of the nodes failed to join the cluster. Upon inspection of /var/log/cassandra/system.log, I found the following:
WARN [GossipStage:1] 2017-03-06 21:06:36,648 TokenMetadata.java:252 - Changing /192.168.211.82's host ID from cff3ef25-9a47-4ea4-9519-b85d20bef3ee to 59f2da9f-0b85-452f-b61a-fa990de53e4b further down: ERROR [main] 2017-03-06 21:20:14,718 CassandraDaemon.java:747 - Exception encountered during startup java.lang.RuntimeException: A node with address /192.168.211.82 already exists, cancelling join. Use cassandra.replace_address if you want to replace this node. at org.apache.cassandra.service.StorageService.checkForEndpointCollision(StorageService.java:491) ~[apache-cassandra-3.9.0.jar:3.9.0] at org.apache.cassandra.service.StorageService.prepareToJoin(StorageService.java:778) ~[apache-cassandra-3.9.0.jar:3.9.0] at org.apache.cassandra.service.StorageService.initServer(StorageService.java:648) ~[apache-cassandra-3.9.0.jar:3.9.0] at org.apache.cassandra.service.StorageService.initServer(StorageService.java:548) ~[apache-cassandra-3.9.0.jar:3.9.0] at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:385) [apache-cassandra-3.9.0.jar:3.9.0] at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:601) [apache-cassandra-3.9.0.jar:3.9.0] at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:730) [apache-cassandra-3.9.0.jar:3.9.0] nodetool status: UN 192.168.211.88 2.58 TiB 256 32.0% 9de2d3ef-5ae1-4c7f-8560-730757a6d1ae rack1 UN 192.168.211.80 2.26 TiB 256 33.9% d83829d3-a1d3-4e6c-b014-7cfe45e22d67 rack1 UN 192.168.211.81 2.91 TiB 256 34.1% 0cafd24e-d3ed-4e51-b586-0b496835a931 rack1 DN 192.168.211.82 551.45 KiB 256 31.9% 59f2da9f-0b85-452f-b61a-fa990de53e4b rack1 UN 192.168.211.83 2.32 TiB 256 32.7% db006e31-03fa-486a-8512-f88eb583bd0c rack1 UN 192.168.211.84 2.54 TiB 256 34.3% a9a50a74-2fc2-4866-a03a-ec95a7866183 rack1 UN 192.168.211.85 2.4 TiB 256 35.9% 733e6703-c18f-432f-a787-3731f80ba42d rack1 UN 192.168.211.86 2.34 TiB 256 32.1% 0daa06fa-708f-4ff8-a15e-861f1a53113a rack1 UN 192.168.211.87 4.07 TiB 256 33.1% 2aa578c6-1332-4b94-81c6-c3ce005a52ef rack1 My questions: 1. Why did the host ID change? 2. If I modify cassandra-env.sh to include JVM_OPTS="$JVM_OPTS -Dcassandra.replace_address=192.168.211.82", will I recover the data on the original node? It is still on the node's hard drive.I really don't want to have to restream 2.6TB of data onto a "new" node.
