Any clue on this ?
2013/2/25 Alain RODRIGUEZ <arodr...@gmail.com> > Hi, > > I am having issues after decommissioning 3 nodes one by one of my 1.1.6 C* > cluster (RF=3): > > On the "c.164" node, which was added a week after removing the 3 nodes, > with gossipinfo I have: > > /a.135 > RPC_ADDRESS:0.0.0.0 > STATUS:NORMAL,127605887595351923798765477786913079296 > RELEASE_VERSION:1.1.6 > RACK:1b > SCHEMA:49aee81e-7c46-31bd-8e4b-dfd07d74d94c > DC:eu-west > LOAD:3.40954135223E11 > /b.173 > RPC_ADDRESS:0.0.0.0 > STATUS:NORMAL,0 > RELEASE_VERSION:1.1.6 > RACK:1b > SCHEMA:49aee81e-7c46-31bd-8e4b-dfd07d74d94c > DC:eu-west > LOAD:3.32757832183E11 > /c.164 > RPC_ADDRESS:0.0.0.0 > STATUS:NORMAL,85070591730234615865843651857942052864 > RELEASE_VERSION:1.1.6 > RACK:1b > SCHEMA:49aee81e-7c46-31bd-8e4b-dfd07d74d94c > DC:eu-west > LOAD:2.93726484252E11 > /d.6 > RPC_ADDRESS:0.0.0.0 > STATUS:NORMAL,42535295865117307932921825928971026432 > RELEASE_VERSION:1.1.6 > RACK:1b > SCHEMA:49aee81e-7c46-31bd-8e4b-dfd07d74d94c > DC:eu-west > LOAD:2.85020693654E11 > > > On the 3 other nodes I see this: > > > /a.135 > RPC_ADDRESS:0.0.0.0 > SCHEMA:49aee81e-7c46-31bd-8e4b-dfd07d74d94c > RELEASE_VERSION:1.1.6 > RACK:1b > STATUS:NORMAL,127605887595351923798765477786913079296 > DC:eu-west > LOAD:3.40974023487E11 > /10.64.167.32 > RPC_ADDRESS:0.0.0.0 > SCHEMA:d9adcce3-09ed-3e7f-a6a3-147d4283ed15 > RELEASE_VERSION:1.1.6 > RACK:1b > STATUS:LEFT,28356863910078203714492389662765613056,1359823927010 > DC:eu-west > LOAD:1.47947624544E11 > /10.250.202.154 > RPC_ADDRESS:0.0.0.0 > SCHEMA:d9adcce3-09ed-3e7f-a6a3-147d4283ed15 > RELEASE_VERSION:1.1.6 > RACK:1b > STATUS:LEFT,85070591730234615865843651857942052863,1359808901882 > DC:eu-west > LOAD:1.45049060742E11 > /b.173 > RPC_ADDRESS:0.0.0.0 > SCHEMA:49aee81e-7c46-31bd-8e4b-dfd07d74d94c > RELEASE_VERSION:1.1.6 > RACK:1b > STATUS:NORMAL,0 > DC:eu-west > LOAD:3.32760540235E11 > /c.164 > RPC_ADDRESS:0.0.0.0 > SCHEMA:49aee81e-7c46-31bd-8e4b-dfd07d74d94c > RELEASE_VERSION:1.1.6 > RACK:1b > LOAD:2.93751485625E11 > DC:eu-west > STATUS:NORMAL,85070591730234615865843651857942052864 > /10.64.103.228 > RPC_ADDRESS:0.0.0.0 > SCHEMA:d9adcce3-09ed-3e7f-a6a3-147d4283ed15 > RELEASE_VERSION:1.1.6 > RACK:1b > STATUS:LEFT,141784319550391032739561396922763706367,1359893766266 > DC:eu-west > LOAD:2.46247802646E11 > /d.6 > RPC_ADDRESS:0.0.0.0 > SCHEMA:49aee81e-7c46-31bd-8e4b-dfd07d74d94c > RELEASE_VERSION:1.1.6 > RACK:1b > STATUS:NORMAL,42535295865117307932921825928971026432 > DC:eu-west > LOAD:2.85042986093E11 > > > Since I removed this 3 nodes (marked as "left") at least 3 weeks ago, > shouldn't gossip have them totally removed for a while ? > > The "c.164", node that doesn't show nodes that left the ring in > gosssipinfo, is logging every minute the following: > > ... > INFO [GossipStage:1] 2013-02-25 10:18:56,269 Gossiper.java (line 830) > InetAddress /10.64.167.32 is now dead. > INFO [GossipStage:1] 2013-02-25 10:18:56,283 Gossiper.java (line 830) > InetAddress /10.250.202.154 is now dead. > INFO [GossipStage:1] 2013-02-25 10:18:56,297 Gossiper.java (line 830) > InetAddress /10.64.103.228 is now dead. > INFO [GossipStage:1] 2013-02-25 10:19:57,700 Gossiper.java (line 830) > InetAddress /10.64.167.32 is now dead. > INFO [GossipStage:1] 2013-02-25 10:19:57,721 Gossiper.java (line 830) > InetAddress /10.250.202.154 is now dead. > INFO [GossipStage:1] 2013-02-25 10:19:57,742 Gossiper.java (line 830) > InetAddress /10.64.103.228 is now dead. > INFO [GossipStage:1] 2013-02-25 10:20:58,722 Gossiper.java (line 830) > InetAddress /10.64.167.32 is now dead. > INFO [GossipStage:1] 2013-02-25 10:20:58,739 Gossiper.java (line 830) > InetAddress /10.250.202.154 is now dead. > INFO [GossipStage:1] 2013-02-25 10:20:58,754 Gossiper.java (line 830) > InetAddress /10.64.103.228 is now dead. > ... > > All this look a bit weird to me, is this normal ? > > Alain > > > >
