[ https://issues.apache.org/jira/browse/CASSANDRA-8318?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14212627#comment-14212627 ]
Brandon Williams edited comment on CASSANDRA-8318 at 11/14/14 6:55 PM: ----------------------------------------------------------------------- bq. The latter is somewhat surprising since that is the IP of the actual replacement node. It doesn't surprise me it can't talk to itself if it hasn't started! That's a perfectly normal message to see during replacement, since we consider the replacee to be 'down' while it's performing the replace. bq. Should I expect it do be there? No, but it shouldn't hurt, either. Numerous bugs around this have been fixed since 2.0.8. What actual token does .51 have? was (Author: brandon.williams): bq. The latter is somewhat surprising since that is the IP of the actual replacement node. It doesn't surprise me it can't talk to itself if it hasn't started! That's a perfectly normal message to see during replacement, since we consider the replacee to be 'down' while it's performing the replace. bq. Should I expect it do be there? No, but it shouldn't hurt, either. Numerous bugs around this have been fixed since 2.0.8. What actualy token does .51 have? > Unable to replace a node > ------------------------ > > Key: CASSANDRA-8318 > URL: https://issues.apache.org/jira/browse/CASSANDRA-8318 > Project: Cassandra > Issue Type: Bug > Environment: 2.0.8.39 (Datastax DSE 4.5.3) > Reporter: Jens Rantil > Attachments: X.X.X.56.log > > > Had a hardware failure of a node. I followed the Datastax documentation[1] on > how to replace the node X.X.X.51 using a brand new node with the same IP. > Since it didn't come up after waiting for ~5 minutes or so, I decided to > replace X.X.X.51 with a brand new unused IP X.X.X.56 instead. It now seems > like my gossip is some weird state. When I start the replacement node I see > line like > {noformat} > INFO [GossipStage:1] 2014-11-14 14:57:03,025 Gossiper.java (line 901) > InetAddress /X.X.X.51 is now DOWN > INFO [GossipStage:1] 2014-11-14 14:57:03,042 Gossiper.java (line 901) > InetAddress /X.X.X.56 is now DOWN > {noformat} > . The latter is somewhat surprising since that is the IP of the actual > replacement node. It doesn't surprise me it can't talk to itself if it hasn't > started! > Eventually the replacement node shuts down with > {noformat} > ERROR [main] 2014-11-14 14:58:06,031 CassandraDaemon.java (line 513) > Exception encountered during startup > java.lang.UnsupportedOperationException: Cannot replace token -2 which does > not exist! > at > org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:782) > at > org.apache.cassandra.service.StorageService.initServer(StorageService.java:614) > at > org.apache.cassandra.service.StorageService.initServer(StorageService.java:503) > at > org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:378) > at com.datastax.bdp.server.DseDaemon.setup(DseDaemon.java:374) > at > org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:496) > at com.datastax.bdp.server.DseDaemon.main(DseDaemon.java:615) > INFO [Thread-2] 2014-11-14 14:58:06,035 DseDaemon.java (line 461) DSE > shutting down... > INFO [StorageServiceShutdownHook] 2014-11-14 14:58:06,037 Gossiper.java > (line 1307) Announcing shutdown > INFO [Thread-2] 2014-11-14 14:58:06,046 PluginManager.java (line 355) All > plugins are stopped. > INFO [Thread-2] 2014-11-14 14:58:06,047 CassandraDaemon.java (line 463) > Cassandra shutting down... > ERROR [Thread-2] 2014-11-14 14:58:06,047 CassandraDaemon.java (line 199) > Exception in thread Thread[Thread-2,5,main] > java.lang.NullPointerException > at > org.apache.cassandra.service.CassandraDaemon.stop(CassandraDaemon.java:464) > at com.datastax.bdp.server.DseDaemon.stop(DseDaemon.java:464) > at com.datastax.bdp.server.DseDaemon$1.run(DseDaemon.java:364){noformat} > All nodes are showing > {noformat} > root@machine-2:~# nodetool status company > Datacenter: Analytics > ===================== > Status=Up/Down > |/ State=Normal/Leaving/Joining/Moving > -- Address Load Tokens Owns (effective) Host ID > Rack > UN X.X.X.50 18.35 GB 1 16.7% > 25efdbcd-14d3-4e9c-803a-3db5795d4efa rack1 > DN X.X.X.51 195.67 KB 1 16.7% > d97cf86f-bfaf-4488-b716-26d71635a8fc rack1 > UN X.X.X.52 18.7 GB 1 16.7% > caa32f68-5a6b-4d87-80bd-baa66a9b61ce rack1 > UN X.X.X.53 18.56 GB 1 16.7% > e219321e-a6d5-48c4-9bad-d2e25429b1d2 rack1 > UN X.X.X.54 19.69 GB 1 16.7% > 3cd36895-ee47-41c1-a5f5-41cb0f8526a6 rack1 > UN X.X.X.55 18.88 GB 1 16.7% > 7d3f73c4-724e-45a6-bcf9-d3072dfc157f rack1 > Datacenter: Cassandra > ===================== > Status=Up/Down > |/ State=Normal/Leaving/Joining/Moving > -- Address Load Tokens Owns (effective) Host ID > Rack > UN X.X.X.33 128.95 GB 256 100.0% > 871968c9-1d6b-4f06-ba90-8b3a8d92dcf0 rack1 > UN X.X.X.32 115.3 GB 256 100.0% > d7cacd89-8613-4de5-8a5e-a2c53c41ea45 rack1 > UN X.X.X.31 130.45 GB 256 100.0% > 48cb0782-6c9a-4805-9330-38e192b6b680 rack1 > {noformat} > , but when X.X.X.56 is starting is shows > {noformat} > root@machine-1:/var/lib/cassandra# nodetool status > Note: Ownership information does not include topology; for complete > information, specify a keyspace > Datacenter: Analytics > ===================== > Status=Up/Down > |/ State=Normal/Leaving/Joining/Moving > -- Address Load Tokens Owns Host ID > Rack > UN X.X.X.50 18.41 GB 1 0.2% 25efdbcd-14d3-4e9c-803a-3db5795d4efa > rack1 > UN X.X.X.52 19.07 GB 1 0.0% caa32f68-5a6b-4d87-80bd-baa66a9b61ce > rack1 > UN X.X.X.53 18.65 GB 1 0.1% e219321e-a6d5-48c4-9bad-d2e25429b1d2 > rack1 > UN X.X.X.54 19.69 GB 1 0.0% 3cd36895-ee47-41c1-a5f5-41cb0f8526a6 > rack1 > UN X.X.X.55 18.97 GB 1 0.2% 7d3f73c4-724e-45a6-bcf9-d3072dfc157f > rack1 > Datacenter: Cassandra > ===================== > Status=Up/Down > |/ State=Normal/Leaving/Joining/Moving > -- Address Load Tokens Owns Host ID > Rack > UN X.X.X.33 129.72 GB 256 21.7% 871968c9-1d6b-4f06-ba90-8b3a8d92dcf0 > rack1 > UN X.X.X.32 116 GB 256 12.4% d7cacd89-8613-4de5-8a5e-a2c53c41ea45 > rack1 > UN X.X.X.31 130.62 GB 256 65.3% 48cb0782-6c9a-4805-9330-38e192b6b680 > rack1 > {noformat} > The above cluster state does not seem to replicate to the rest of the cluster > (hasn't so far). > Any input on how I can restore world order is appreciated. > [1] > http://www.datastax.com/documentation/cassandra/2.0/cassandra/operations/ops_replace_node_t.html -- This message was sent by Atlassian JIRA (v6.3.4#6332)