Prateek Agarwal created CASSANDRA-12560: -------------------------------------------
Summary: Cassandra Restart issues while restoring to a new cluster Key: CASSANDRA-12560 URL: https://issues.apache.org/jira/browse/CASSANDRA-12560 Project: Cassandra Issue Type: Bug Components: Configuration Environment: distro: Ubuntu 14.04 LTS Reporter: Prateek Agarwal I am restoring to a fresh new Cassandra 2.2.5 cluster consisting of 3 nodes. Initial cluster health of the NEW cluster: {code} -- Address Load Tokens Owns Host ID Rack UN 10.40.1.1 259.31 KB 256 ? d2b29b08-9eac-4733-9798-019275d66cfc uswest1adevc UN 10.40.1.2 230.12 KB 256 ? 5484ab11-32b1-4d01-a5fe-c996a63108f1 uswest1adevc UN 10.40.1.3 248.47 KB 256 ? bad95fe2-70c5-4a2f-b517-d7fd7a32bc45 uswest1cdevc {code} As part of the [restore instructions in Datastax 2.2 docs|http://docs.datastax.com/en/cassandra/2.2/cassandra/operations/opsSnapshotRestoreNewCluster.html], i do the following on the new cluster: 1) cassandra stop on all of the three nodes one by one. 2) Edit cassandra.yaml for all of the three nodes with the backup'ed token ring information. [Step 2 from docs] 3) Remove the contents from /var/lib/cassandra/data/system/* [Step 4 from docs] 4) cassandra start on nodes 10.40.1.1, 10.40.1.2, 10.40.1.3 respectively. Result: 10.40.1.1 restarts back successfully: {code} -- Address Load Tokens Owns Host ID Rack UN 10.40.1.1 259.31 KB 256 ? 2d23add3-9eac-4733-9798-019275d125d3 uswest1adevc {code} But the second and the third nodes fail to restart stating: {code} java.lang.RuntimeException: A node with address 10.40.1.2 already exists, cancelling join. Use cassandra.replace_address if you want to replace this node. at org.apache.cassandra.service.StorageService.checkForEndpointCollision(StorageService.java:546) ~[apache-cassandra-2.2.5.jar:2.2.5] at org.apache.cassandra.service.StorageService.prepareToJoin(StorageService.java:766) ~[apache-cassandra-2.2.5.jar:2.2.5] at org.apache.cassandra.service.StorageService.initServer(StorageService.java:693) ~[apache-cassandra-2.2.5.jar:2.2.5] at org.apache.cassandra.service.StorageService.initServer(StorageService.java:585) ~[apache-cassandra-2.2.5.jar:2.2.5] at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:300) [apache-cassandra-2.2.5.jar:2.2.5] at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:516) [apache-cassandra-2.2.5.jar:2.2.5] at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:625) [apache-cassandra-2.2.5.jar:2.2.5] INFO [StorageServiceShutdownHook] 2016-08-09 18:13:21,980 Gossiper.java:1449 - Announcing shutdown {code} {code} java.lang.RuntimeException: A node with address 10.40.1.3 already exists, cancelling join. Use cassandra.replace_address if you want to replace this node. ... {code} Eventual cluster health: {code} -- Address Load Tokens Owns Host ID Rack UN 10.40.1.1 259.31 KB 256 ? 2d23add3-9eac-4733-9798-019275d125d3 uswest1adevc DN 10.40.1.2 230.12 KB 256 ? 6w2321ad-32b1-4d01-a5fe-c996a63108f1 uswest1adevc DN 10.40.1.3 248.47 KB 256 ? 9et4944d-70c5-4a2f-b517-d7fd7a32bc45 uswest1cdevc {code} I understand that the HostID of a node might change after system dirs are removed. I think the restore docs are incomplete and need to mention the 'replace IP' part as well OR am i missing something in my steps? -- This message was sent by Atlassian JIRA (v6.3.4#6332)