thanks a lot Alian. We did rely on "unsafeassasinate" earlier, which worked. We were planning to upgrade from 2.0.14 version to 2.1.12, on all our clusters. But we are trying to figure out why decommissioned nodes are showing up in the "nodetool describecluster" as "UNREACHABLE".
thanks Sai On Wed, Feb 17, 2016 at 5:42 AM, Alain RODRIGUEZ <arodr...@gmail.com> wrote: > Hi, > > nodetool gossipinfo shows the decommissioned nodes as "LEFT" > > > I believe this is the expected behavior, we keep some a trace of leaving > nodes for a few days, this shouldn't be an issue for you > > nodetool describecluster shows the decommissioned nodes as UNREACHABLE. >> > > This is a weird behaviour I haven't see for a while. You might want to dig > this some more. > > Restarting the entire cluster, everytime a node is decommissioned does >> not seem right >> > > Meanwhile, if you are sure the node is out and streams have ended, I guess > it could be ok to use a JMX client (MX4J, JConsole...) and then use the JMX > method Gossiper.unsafeAssassinateEndpoints(ip_address) to assassinate the > gone node from any of the remaining nodes. > > How to --> > http://tumblr.doki-pen.org/post/22654515359/assassinating-cassandra-nodes > (3 years old post, I partially read it, but I think it might still be > relevant) > > Has anybody experienced similar behaviour > > > FTR, 3 years old similar issue I faced --> > http://grokbase.com/t/cassandra/user/127knx7nn0/unreachable-node-not-in-nodetool-ring > > FWIW, people using C* = 3.x, this is exposed through nodetool --> > https://docs.datastax.com/en/cassandra/3.x/cassandra/tools/toolsAssassinate.html > > Keep in mind that something called 'unsafe' and 'assassinate' at the same > time is not something you want to use in a regular decommissioning process > as it drop the node with no file transfer, you basically totally lose a > node (unless node is out already which seems to be your case, it should be > safe to use it in your case). I only used it to fix gossip status in the > past or at some point when forcing a removenode was not working, followed > by full repairs on remaining nodes. > > C*heers, > ----------------- > Alain Rodriguez > France > > The Last Pickle > http://www.thelastpickle.com > > 2016-02-16 20:08 GMT+01:00 sai krishnam raju potturi <pskraj...@gmail.com> > : > >> hi; >> we have a 12 node cluster across 2 datacenters. We are currently >> using cassandra 2.1.12 version. >> >> SNITCH : GossipingPropertyFileSnitch >> >> When we decommissioned few nodes in a particular datacenter and observed >> the following : >> >> nodetool status shows only the live nodes in the cluster. >> >> nodetool describecluster shows the decommissioned nodes as UNREACHABLE. >> >> nodetool gossipinfo shows the decommissioned nodes as "LEFT" >> >> >> When the live nodes were restarted, "nodetool describecluster" shows >> only the live nodes, which is expected. >> >> Purging the gossip info too did not help. >> >> INFO 17:27:07 InetAddress /X.X.X.X is now DOWN >> INFO 17:27:07 Removing tokens [125897680671740685543105407593050165202, >> 140213388002871593911508364312533329916, >> 98576967436431350637134234839492449485] for /X.X.X.X >> INFO 17:27:07 InetAddress /X.X.X.X is now DOWN >> INFO 17:27:07 Removing tokens [11116977666116265389022494863106850615, >> 111270759969411259938117902792984586225, >> 138611464975439236357814418845450428175] for /X.X.X.X >> >> Has anybody experienced similar behaviour. Restarting the entire cluster, >> everytime a node is decommissioned does not seem right. Thanks in advance >> for the help. >> >> >> thanks >> Sai >> >> >> >
