I recently removed a node (with decommission) from our cluster.
I added a couple new nodes and am now trying to rebalance the cluster using
nodetool move.
However, netstats shows that the node being "moved" is trying to stream data
to the node that I already decommissioned yesterday.
The removed node was powered-off, taken out of dns, its IP is not even
pingable. It was never a seed neither.
This is cassandra 0.7.5 on 64bit linux. How do I tell the cluster that this
node is gone? Gossip should have detected this. The ring commands shows the
correct cluster IPs.
Here is a portion of netstats. 10.46.108.102 is the node which was removed.
Mode: Leaving: streaming data to other nodes
Streaming to: /10.46.108.102
/var/lib/cassandra/data/DFS/main-f-1064-Data.db/(4681027,5195491),(5195491,15308570),(15308570,15891710),(16336750,20558705),(20558705,29112203),(29112203,36279329),(36465942,36623223),(36740457,37227058),(37227058,42206994),(42206994,47380294),(47635053,47709813),(47709813,48353944),(48621287,49406499),(53330048,53571312),(53571312,54153922),(54153922,59857615),(59857615,61029910),(61029910,61871509),(62190800,62498605),(62824281,62964830),(63511604,64353114),(64353114,64760400),(65174702,65919771),(65919771,66435630),(81440029,81725949),(81725949,83313847),(83313847,83908709),(88983863,89237303),(89237303,89934199),(89934199,97
...................
5693491,14795861666),(14795861666,14796105318),(14796105318,14796366886),(14796699825,14803874941),(14803874941,14808898331),(14808898331,14811670699),(14811670699,14815125177),(14815125177,14819765003),(14820229433,14820858266)
progress=280574376402/12434049900 - 2256%
.....
Note 10.46.108.102 is NOT part of the ring.
Address Status State Load Owns Token
148873535527910577765226390751398592512
10.46.108.100 Up Normal 71.73 GB 12.50% 0
10.46.108.101 Up Normal 109.69 GB 12.50%
21267647932558653966460912964485513216
10.47.108.100 Up Leaving 281.95 GB 37.50%
85070591730234615865843651857942052863 <----- currently being "moved"
10.47.108.102 Up Normal 210.77 GB 0.00%
85070591730234615865843651857942052864
10.47.108.101 Up Normal 289.59 GB 16.67%
113427455640312821154458202477256070484
10.46.108.103 Up Normal 299.87 GB 8.33%
127605887595351923798765477786913079296
10.47.108.103 Up Normal 94.99 GB 12.50%
148873535527910577765226390751398592511
10.46.108.104 Up Normal 103.01 GB 0.00%
148873535527910577765226390751398592512
