Hello,
Short story:
- I had to replace a dead node in my cluster
- 1 week after, dead node is still seen as DN by 3 out of 5 nodes
- dead node has null host_id
- assassinate on dead node fails with error
How can I get rid of this dead node ?
Long story:
I had a 3 nodes cluster (Cassandra 3.9) ; one node went dead. I built a
new node from scratch and "replaced" the dead node using the information
from this page
https://docs.datastax.com/en/cassandra/3.0/cassandra/operations/opsReplaceNode.html.
It looked like the replacement went ok.
I added two more nodes to strengthen the cluster.
A few days have passed and the dead node is still visible and marked as
"down" on 3 of 5 nodes in nodetool status:
-- Address Load Tokens Owns (effective) Host ID
Rack
UN 192.168.1.9 16 GiB 256 35.0%
76223d4c-9d9f-417f-be27-cebb791cddcc rack1
UN 192.168.1.12 16.09 GiB 256 34.0%
719601e2-54a6-440e-a379-c9cf2dc20564 rack1
UN 192.168.1.14 14.16 GiB 256 32.6%
d8017a03-7e4e-47b7-89b9-cd9ec472d74f rack1
UN 192.168.1.17 15.4 GiB 256 34.1%
fa238b21-1db1-47dc-bfb7-beedc6c9967a rack1
DN 192.168.1.18 24.3 GiB 256 33.7% null
rack1
UN 192.168.1.22 19.06 GiB 256 30.7%
09d24557-4e98-44c3-8c9d-53c4c31066e1 rack1
Its host ID is null, so I cannot use nodetool removenode. Moreover
nodetool assassinate 192.168.1.18 fails with :
error: null
-- StackTrace --
java.lang.NullPointerException
And in system.log:
INFO [RMI TCP Connection(16)-127.0.0.1] 2019-03-27 17:39:38,595
Gossiper.java:585 - Sleeping for 30000ms to ensure /192.168.1.18 does
not change
INFO [CompactionExecutor:547] 2019-03-27 17:39:38,669
AutoSavingCache.java:393 - Saved KeyCache (27316 items) in 163 ms
INFO [IndexSummaryManager:1] 2019-03-27 17:40:03,620
IndexSummaryRedistribution.java:75 - Redistributing index summaries
INFO [RMI TCP Connection(16)-127.0.0.1] 2019-03-27 17:40:08,597
Gossiper.java:1029 - InetAddress /192.168.1.18 is now DOWN
INFO [RMI TCP Connection(16)-127.0.0.1] 2019-03-27 17:40:08,599
StorageService.java:2324 - Removing tokens [-1061369577393671924,...]
ERROR [GossipStage:1] 2019-03-27 17:40:08,600 CassandraDaemon.java:226 -
Exception in thread Thread[GossipStage:1,5,main]
java.lang.NullPointerException: null
In system.peers, the dead node shows and has the same ID as the
replacing node :
cqlsh> select peer, host_id from system.peers;
peer | host_id
--------------+--------------------------------------
192.168.1.18 | 09d24557-4e98-44c3-8c9d-53c4c31066e1
192.168.1.22 | 09d24557-4e98-44c3-8c9d-53c4c31066e1
192.168.1.9 | 76223d4c-9d9f-417f-be27-cebb791cddcc
192.168.1.14 | d8017a03-7e4e-47b7-89b9-cd9ec472d74f
192.168.1.12 | 719601e2-54a6-440e-a379-c9cf2dc20564
Dead node and replacing node have different tokens in system.peers.
I should add that I also tried decommission on a node that still
192.168.1.18 in its peers. - it is still marked as "leaving" 5 days
later. Nothing in notetool netstats or nodetool compactionstats.
Thank you for taking the time to read this. Hope you can help.
Alex
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
For additional commands, e-mail: user-h...@cassandra.apache.org
