[jira] [Created] (CASSANDRA-8422) cassandra won't start up due to Unable to gossip with any seeds on the decommissioned node
Masashi Ozawa created CASSANDRA-8422: Summary: cassandra won't start up due to Unable to gossip with any seeds on the decommissioned node Key: CASSANDRA-8422 URL: https://issues.apache.org/jira/browse/CASSANDRA-8422 Project: Cassandra Issue Type: Bug Components: Core Reporter: Masashi Ozawa - 2-node * nodeA - seed * nodeB 1. decommission nodeB from the cluster with nodetool when it's finished, kill cassandra process on nodeB 2. delete data from commit/cache/data directories on nodeB 3. try to start cassandra on nodeB (first time) = FAILED with Unable to gossip with any seeds 4. try to start cassandra on nodeB (second time) = OK It was not a one-time shot. I tried it a several times and encountered the same issue for some reason. ERROR [main] 2014-11-27 18:44:55,017 CassandraDaemon.java (line 513) Exception encountered during startup java.lang.RuntimeException: Unable to gossip with any seeds at org.apache.cassandra.gms.Gossiper.doShadowRound(Gossiper.java:1211) at org.apache.cassandra.service.StorageService.checkForEndpointCollision(StorageService.java:445) at org.apache.cassandra.service.StorageService.prepareToJoin(StorageService.java:659) at org.apache.cassandra.service.StorageService.initServer(StorageService.java:611) at org.apache.cassandra.service.StorageService.initServer(StorageService.java:503) at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:378) at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:496) at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:585) INFO [StorageServiceShutdownHook] 2014-11-27 18:44:55,076 Gossiper.java (line 1307) Announcing shutdown For the first time, it looks like that the recommission node(goo184) did not receive a GossipDigestAckMessage from a seed(test130) ? Don't know why. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8422) cassandra won't start up due to Unable to gossip with any seeds on the decommissioned node
[ https://issues.apache.org/jira/browse/CASSANDRA-8422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14234085#comment-14234085 ] Masashi Ozawa commented on CASSANDRA-8422: -- The similar issue has been reported in CASSANDRA-7350 and it says that this has been fixed in CASSANDRA-6523 in 2.0.9. However it's still happening on my env for some reason. cassandra won't start up due to Unable to gossip with any seeds on the decommissioned node Key: CASSANDRA-8422 URL: https://issues.apache.org/jira/browse/CASSANDRA-8422 Project: Cassandra Issue Type: Bug Components: Core Reporter: Masashi Ozawa - 2-node * nodeA - seed * nodeB 1. decommission nodeB from the cluster with nodetool when it's finished, kill cassandra process on nodeB 2. delete data from commit/cache/data directories on nodeB 3. try to start cassandra on nodeB (first time) = FAILED with Unable to gossip with any seeds 4. try to start cassandra on nodeB (second time) = OK It was not a one-time shot. I tried it a several times and encountered the same issue for some reason. ERROR [main] 2014-11-27 18:44:55,017 CassandraDaemon.java (line 513) Exception encountered during startup java.lang.RuntimeException: Unable to gossip with any seeds at org.apache.cassandra.gms.Gossiper.doShadowRound(Gossiper.java:1211) at org.apache.cassandra.service.StorageService.checkForEndpointCollision(StorageService.java:445) at org.apache.cassandra.service.StorageService.prepareToJoin(StorageService.java:659) at org.apache.cassandra.service.StorageService.initServer(StorageService.java:611) at org.apache.cassandra.service.StorageService.initServer(StorageService.java:503) at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:378) at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:496) at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:585) INFO [StorageServiceShutdownHook] 2014-11-27 18:44:55,076 Gossiper.java (line 1307) Announcing shutdown For the first time, it looks like that the recommission node(goo184) did not receive a GossipDigestAckMessage from a seed(test130) ? Don't know why. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (CASSANDRA-8422) cassandra won't start up due to Unable to gossip with any seeds on the decommissioned node
[ https://issues.apache.org/jira/browse/CASSANDRA-8422?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Masashi Ozawa updated CASSANDRA-8422: - Description: - 2-node * nodeA - seed * nodeB 1. decommission nodeB from the cluster with nodetool when it's finished, kill cassandra process on nodeB 2. delete data from commit/cache/data directories on nodeB 3. try to start cassandra on nodeB (first time) = FAILED with Unable to gossip with any seeds 4. try to start cassandra on nodeB (second time) = OK It was not a one-time shot. I tried it a several times and encountered the same issue for some reason. ERROR [main] 2014-11-27 18:44:55,017 CassandraDaemon.java (line 513) Exception encountered during startup java.lang.RuntimeException: Unable to gossip with any seeds at org.apache.cassandra.gms.Gossiper.doShadowRound(Gossiper.java:1211) at org.apache.cassandra.service.StorageService.checkForEndpointCollision(StorageService.java:445) at org.apache.cassandra.service.StorageService.prepareToJoin(StorageService.java:659) at org.apache.cassandra.service.StorageService.initServer(StorageService.java:611) at org.apache.cassandra.service.StorageService.initServer(StorageService.java:503) at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:378) at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:496) at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:585) INFO [StorageServiceShutdownHook] 2014-11-27 18:44:55,076 Gossiper.java (line 1307) Announcing shutdown was: - 2-node * nodeA - seed * nodeB 1. decommission nodeB from the cluster with nodetool when it's finished, kill cassandra process on nodeB 2. delete data from commit/cache/data directories on nodeB 3. try to start cassandra on nodeB (first time) = FAILED with Unable to gossip with any seeds 4. try to start cassandra on nodeB (second time) = OK It was not a one-time shot. I tried it a several times and encountered the same issue for some reason. ERROR [main] 2014-11-27 18:44:55,017 CassandraDaemon.java (line 513) Exception encountered during startup java.lang.RuntimeException: Unable to gossip with any seeds at org.apache.cassandra.gms.Gossiper.doShadowRound(Gossiper.java:1211) at org.apache.cassandra.service.StorageService.checkForEndpointCollision(StorageService.java:445) at org.apache.cassandra.service.StorageService.prepareToJoin(StorageService.java:659) at org.apache.cassandra.service.StorageService.initServer(StorageService.java:611) at org.apache.cassandra.service.StorageService.initServer(StorageService.java:503) at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:378) at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:496) at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:585) INFO [StorageServiceShutdownHook] 2014-11-27 18:44:55,076 Gossiper.java (line 1307) Announcing shutdown For the first time, it looks like that the recommission node(goo184) did not receive a GossipDigestAckMessage from a seed(test130) ? Don't know why. cassandra won't start up due to Unable to gossip with any seeds on the decommissioned node Key: CASSANDRA-8422 URL: https://issues.apache.org/jira/browse/CASSANDRA-8422 Project: Cassandra Issue Type: Bug Components: Core Reporter: Masashi Ozawa - 2-node * nodeA - seed * nodeB 1. decommission nodeB from the cluster with nodetool when it's finished, kill cassandra process on nodeB 2. delete data from commit/cache/data directories on nodeB 3. try to start cassandra on nodeB (first time) = FAILED with Unable to gossip with any seeds 4. try to start cassandra on nodeB (second time) = OK It was not a one-time shot. I tried it a several times and encountered the same issue for some reason. ERROR [main] 2014-11-27 18:44:55,017 CassandraDaemon.java (line 513) Exception encountered during startup java.lang.RuntimeException: Unable to gossip with any seeds at org.apache.cassandra.gms.Gossiper.doShadowRound(Gossiper.java:1211) at org.apache.cassandra.service.StorageService.checkForEndpointCollision(StorageService.java:445) at org.apache.cassandra.service.StorageService.prepareToJoin(StorageService.java:659) at org.apache.cassandra.service.StorageService.initServer(StorageService.java:611) at org.apache.cassandra.service.StorageService.initServer(StorageService.java:503) at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:378) at
[jira] [Commented] (CASSANDRA-7350) Decommissioning nodes borks the seed node - can't add additional nodes
[ https://issues.apache.org/jira/browse/CASSANDRA-7350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14234939#comment-14234939 ] Masashi Ozawa commented on CASSANDRA-7350: -- It's still happening in 2.0.11 for me. Filed CASSANDRA-8422. Decommissioning nodes borks the seed node - can't add additional nodes -- Key: CASSANDRA-7350 URL: https://issues.apache.org/jira/browse/CASSANDRA-7350 Project: Cassandra Issue Type: Bug Components: Core Environment: Ubuntu using the auto-clustering AMI Reporter: Steven Lowenthal Assignee: Shawn Kumar Priority: Minor Labels: qa-resolved Fix For: 2.0.9 1) Launch a 4 node cluster - I used the auto-clustering AMI (you get nodes 0-3) 2) decommission that last 2 nodes (nodes , leaving a 2 node cluster) 3) wipe the data directories from node 2 4) bootstrap node2 - it won't join unable to gossip with any seeds. If you bootstrap the node a second time, it will join. However if you try to bootstrap node 3, it will also fail. I discovered that bouncing the seed node fixes the problem. I think it cropped up in 2.0.7. Error: ERROR [main] 2014-06-03 21:52:46,649 CassandraDaemon.java (line 497) Exception encountered during startup java.lang.RuntimeException: Unable to gossip with any seeds at org.apache.cassandra.gms.Gossiper.doShadowRound(Gossiper.java:1193) at org.apache.cassandra.service.StorageService.checkForEndpointCollision(StorageService.java:447) at org.apache.cassandra.service.StorageService.prepareToJoin(StorageService.java:656) at org.apache.cassandra.service.StorageService.initServer(StorageService.java:612) at org.apache.cassandra.service.StorageService.initServer(StorageService.java:505) at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:362) at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:480) at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:569) ERROR [StorageServiceShutdownHook] 2014-06-03 21:52:46,741 CassandraDaemon.java (line 198) Exception in thread Thread[StorageServi -- This message was sent by Atlassian JIRA (v6.3.4#6332)
