[jira] [Commented] (CASSANDRA-10298) Replaced dead node stayed in gossip forever

2015-09-10 Thread Michael Shuler (JIRA) 

[ 
https://issues.apache.org/jira/browse/CASSANDRA-10298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14739197#comment-14739197
 ] 

Michael Shuler commented on CASSANDRA-10298:


What steps did you take to remove the node from the cluster?

> Replaced dead node stayed in gossip forever
> ---
>
> Key: CASSANDRA-10298
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10298
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Dikang Gu
> Attachments: CASSANDRA-10298.patch
>
>
> The dead node stayed in the nodetool status,
> DN  10.210.165.55379.76 GB  256 ?   null
> And in the log, it throws NPE when trying to remove it.
> 2015-09-10_06:41:22.92453 ERROR 06:41:22 Exception in thread 
> Thread[GossipStage:1,5,main]
> 2015-09-10_06:41:22.92454 java.lang.NullPointerException: null
> 2015-09-10_06:41:22.92455   at 
> org.apache.cassandra.utils.UUIDGen.decompose(UUIDGen.java:100) 
> 2015-09-10_06:41:22.92455   at 
> org.apache.cassandra.db.HintedHandOffManager.deleteHintsForEndpoint(HintedHandOffManager.java:201)
>  
> 2015-09-10_06:41:22.92455   at 
> org.apache.cassandra.service.StorageService.excise(StorageService.java:1886) 
> 2015-09-10_06:41:22.92455   at 
> org.apache.cassandra.service.StorageService.excise(StorageService.java:1902) 
> 2015-09-10_06:41:22.92456   at 
> org.apache.cassandra.service.StorageService.handleStateLeft(StorageService.java:1805)
> 2015-09-10_06:41:22.92457   at 
> org.apache.cassandra.service.StorageService.onChange(StorageService.java:1473)
>  
> 2015-09-10_06:41:22.92457   at 
> org.apache.cassandra.service.StorageService.onJoin(StorageService.java:2099) 
> 2015-09-10_06:41:22.92457   at 
> org.apache.cassandra.gms.Gossiper.handleMajorStateChange(Gossiper.java:1009) 
> 2015-09-10_06:41:22.92458   at 
> org.apache.cassandra.gms.Gossiper.applyStateLocally(Gossiper.java:1085) 
> 2015-09-10_06:41:22.92458   at 
> org.apache.cassandra.gms.GossipDigestAck2VerbHandler.doVerb(GossipDigestAck2VerbHandler.java:49)
>  
> 2015-09-10_06:41:22.92458   at 
> org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:62) 
> 2015-09-10_06:41:22.92459   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>  ~[na:1.7.0_45]
> 2015-09-10_06:41:22.92460   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>  ~[na:1.7.0_45]



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-10298) Replaced dead node stayed in gossip forever

2015-09-10 Thread Dikang Gu (JIRA) 

[ 
https://issues.apache.org/jira/browse/CASSANDRA-10298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14740275#comment-14740275
 ] 

Dikang Gu commented on CASSANDRA-10298:
---

[~mshuler] I use the -Dcassandra.replace_address= to replace that 
dead node.

> Replaced dead node stayed in gossip forever
> ---
>
> Key: CASSANDRA-10298
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10298
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Dikang Gu
> Attachments: CASSANDRA-10298.patch
>
>
> The dead node stayed in the nodetool status,
> DN  10.210.165.55379.76 GB  256 ?   null
> And in the log, it throws NPE when trying to remove it.
> 2015-09-10_06:41:22.92453 ERROR 06:41:22 Exception in thread 
> Thread[GossipStage:1,5,main]
> 2015-09-10_06:41:22.92454 java.lang.NullPointerException: null
> 2015-09-10_06:41:22.92455   at 
> org.apache.cassandra.utils.UUIDGen.decompose(UUIDGen.java:100) 
> 2015-09-10_06:41:22.92455   at 
> org.apache.cassandra.db.HintedHandOffManager.deleteHintsForEndpoint(HintedHandOffManager.java:201)
>  
> 2015-09-10_06:41:22.92455   at 
> org.apache.cassandra.service.StorageService.excise(StorageService.java:1886) 
> 2015-09-10_06:41:22.92455   at 
> org.apache.cassandra.service.StorageService.excise(StorageService.java:1902) 
> 2015-09-10_06:41:22.92456   at 
> org.apache.cassandra.service.StorageService.handleStateLeft(StorageService.java:1805)
> 2015-09-10_06:41:22.92457   at 
> org.apache.cassandra.service.StorageService.onChange(StorageService.java:1473)
>  
> 2015-09-10_06:41:22.92457   at 
> org.apache.cassandra.service.StorageService.onJoin(StorageService.java:2099) 
> 2015-09-10_06:41:22.92457   at 
> org.apache.cassandra.gms.Gossiper.handleMajorStateChange(Gossiper.java:1009) 
> 2015-09-10_06:41:22.92458   at 
> org.apache.cassandra.gms.Gossiper.applyStateLocally(Gossiper.java:1085) 
> 2015-09-10_06:41:22.92458   at 
> org.apache.cassandra.gms.GossipDigestAck2VerbHandler.doVerb(GossipDigestAck2VerbHandler.java:49)
>  
> 2015-09-10_06:41:22.92458   at 
> org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:62) 
> 2015-09-10_06:41:22.92459   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>  ~[na:1.7.0_45]
> 2015-09-10_06:41:22.92460   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>  ~[na:1.7.0_45]



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-10298) Replaced dead node stayed in gossip forever

2015-09-10 Thread Dikang Gu (JIRA) 

[ 
https://issues.apache.org/jira/browse/CASSANDRA-10298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14740279#comment-14740279
 ] 

Dikang Gu commented on CASSANDRA-10298:
---

The new node successfully finished the replacement, but for some reason, the 
dead node still stayed in the nodetool status for some other live nodes.

I can not remove it from the gossip, even after restarting some live node.

> Replaced dead node stayed in gossip forever
> ---
>
> Key: CASSANDRA-10298
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10298
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Dikang Gu
> Attachments: CASSANDRA-10298.patch
>
>
> The dead node stayed in the nodetool status,
> DN  10.210.165.55379.76 GB  256 ?   null
> And in the log, it throws NPE when trying to remove it.
> 2015-09-10_06:41:22.92453 ERROR 06:41:22 Exception in thread 
> Thread[GossipStage:1,5,main]
> 2015-09-10_06:41:22.92454 java.lang.NullPointerException: null
> 2015-09-10_06:41:22.92455   at 
> org.apache.cassandra.utils.UUIDGen.decompose(UUIDGen.java:100) 
> 2015-09-10_06:41:22.92455   at 
> org.apache.cassandra.db.HintedHandOffManager.deleteHintsForEndpoint(HintedHandOffManager.java:201)
>  
> 2015-09-10_06:41:22.92455   at 
> org.apache.cassandra.service.StorageService.excise(StorageService.java:1886) 
> 2015-09-10_06:41:22.92455   at 
> org.apache.cassandra.service.StorageService.excise(StorageService.java:1902) 
> 2015-09-10_06:41:22.92456   at 
> org.apache.cassandra.service.StorageService.handleStateLeft(StorageService.java:1805)
> 2015-09-10_06:41:22.92457   at 
> org.apache.cassandra.service.StorageService.onChange(StorageService.java:1473)
>  
> 2015-09-10_06:41:22.92457   at 
> org.apache.cassandra.service.StorageService.onJoin(StorageService.java:2099) 
> 2015-09-10_06:41:22.92457   at 
> org.apache.cassandra.gms.Gossiper.handleMajorStateChange(Gossiper.java:1009) 
> 2015-09-10_06:41:22.92458   at 
> org.apache.cassandra.gms.Gossiper.applyStateLocally(Gossiper.java:1085) 
> 2015-09-10_06:41:22.92458   at 
> org.apache.cassandra.gms.GossipDigestAck2VerbHandler.doVerb(GossipDigestAck2VerbHandler.java:49)
>  
> 2015-09-10_06:41:22.92458   at 
> org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:62) 
> 2015-09-10_06:41:22.92459   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>  ~[na:1.7.0_45]
> 2015-09-10_06:41:22.92460   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>  ~[na:1.7.0_45]



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-10298) Replaced dead node stayed in gossip forever

2015-09-14 Thread Jim Witschey (JIRA) 

[ 
https://issues.apache.org/jira/browse/CASSANDRA-10298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14744236#comment-14744236
 ] 

Jim Witschey commented on CASSANDRA-10298:
--

[~dikanggu] Is this related to CASSANDRA-10321, and if so, how? You mentioned 
[in this 
comment|https://issues.apache.org/jira/browse/CASSANDRA-10321?focusedCommentId=14744120&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14744120]
 that the node was running, but marked as having Thrift and Gossip unavailable. 
Is that the same node you tried to remove? and could that unavailable state be 
the result of the failed removal?

> Replaced dead node stayed in gossip forever
> ---
>
> Key: CASSANDRA-10298
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10298
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Dikang Gu
> Attachments: CASSANDRA-10298.patch
>
>
> The dead node stayed in the nodetool status,
> DN  10.210.165.55379.76 GB  256 ?   null
> And in the log, it throws NPE when trying to remove it.
> 2015-09-10_06:41:22.92453 ERROR 06:41:22 Exception in thread 
> Thread[GossipStage:1,5,main]
> 2015-09-10_06:41:22.92454 java.lang.NullPointerException: null
> 2015-09-10_06:41:22.92455   at 
> org.apache.cassandra.utils.UUIDGen.decompose(UUIDGen.java:100) 
> 2015-09-10_06:41:22.92455   at 
> org.apache.cassandra.db.HintedHandOffManager.deleteHintsForEndpoint(HintedHandOffManager.java:201)
>  
> 2015-09-10_06:41:22.92455   at 
> org.apache.cassandra.service.StorageService.excise(StorageService.java:1886) 
> 2015-09-10_06:41:22.92455   at 
> org.apache.cassandra.service.StorageService.excise(StorageService.java:1902) 
> 2015-09-10_06:41:22.92456   at 
> org.apache.cassandra.service.StorageService.handleStateLeft(StorageService.java:1805)
> 2015-09-10_06:41:22.92457   at 
> org.apache.cassandra.service.StorageService.onChange(StorageService.java:1473)
>  
> 2015-09-10_06:41:22.92457   at 
> org.apache.cassandra.service.StorageService.onJoin(StorageService.java:2099) 
> 2015-09-10_06:41:22.92457   at 
> org.apache.cassandra.gms.Gossiper.handleMajorStateChange(Gossiper.java:1009) 
> 2015-09-10_06:41:22.92458   at 
> org.apache.cassandra.gms.Gossiper.applyStateLocally(Gossiper.java:1085) 
> 2015-09-10_06:41:22.92458   at 
> org.apache.cassandra.gms.GossipDigestAck2VerbHandler.doVerb(GossipDigestAck2VerbHandler.java:49)
>  
> 2015-09-10_06:41:22.92458   at 
> org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:62) 
> 2015-09-10_06:41:22.92459   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>  ~[na:1.7.0_45]
> 2015-09-10_06:41:22.92460   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>  ~[na:1.7.0_45]



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-10298) Replaced dead node stayed in gossip forever

2015-09-14 Thread Dikang Gu (JIRA) 

[ 
https://issues.apache.org/jira/browse/CASSANDRA-10298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14744325#comment-14744325
 ] 

Dikang Gu commented on CASSANDRA-10298:
---

[~mambocab], no they are complaining different nodes, so it looks like 
different issues to me.

> Replaced dead node stayed in gossip forever
> ---
>
> Key: CASSANDRA-10298
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10298
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Dikang Gu
> Attachments: CASSANDRA-10298.patch
>
>
> The dead node stayed in the nodetool status,
> DN  10.210.165.55379.76 GB  256 ?   null
> And in the log, it throws NPE when trying to remove it.
> 2015-09-10_06:41:22.92453 ERROR 06:41:22 Exception in thread 
> Thread[GossipStage:1,5,main]
> 2015-09-10_06:41:22.92454 java.lang.NullPointerException: null
> 2015-09-10_06:41:22.92455   at 
> org.apache.cassandra.utils.UUIDGen.decompose(UUIDGen.java:100) 
> 2015-09-10_06:41:22.92455   at 
> org.apache.cassandra.db.HintedHandOffManager.deleteHintsForEndpoint(HintedHandOffManager.java:201)
>  
> 2015-09-10_06:41:22.92455   at 
> org.apache.cassandra.service.StorageService.excise(StorageService.java:1886) 
> 2015-09-10_06:41:22.92455   at 
> org.apache.cassandra.service.StorageService.excise(StorageService.java:1902) 
> 2015-09-10_06:41:22.92456   at 
> org.apache.cassandra.service.StorageService.handleStateLeft(StorageService.java:1805)
> 2015-09-10_06:41:22.92457   at 
> org.apache.cassandra.service.StorageService.onChange(StorageService.java:1473)
>  
> 2015-09-10_06:41:22.92457   at 
> org.apache.cassandra.service.StorageService.onJoin(StorageService.java:2099) 
> 2015-09-10_06:41:22.92457   at 
> org.apache.cassandra.gms.Gossiper.handleMajorStateChange(Gossiper.java:1009) 
> 2015-09-10_06:41:22.92458   at 
> org.apache.cassandra.gms.Gossiper.applyStateLocally(Gossiper.java:1085) 
> 2015-09-10_06:41:22.92458   at 
> org.apache.cassandra.gms.GossipDigestAck2VerbHandler.doVerb(GossipDigestAck2VerbHandler.java:49)
>  
> 2015-09-10_06:41:22.92458   at 
> org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:62) 
> 2015-09-10_06:41:22.92459   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>  ~[na:1.7.0_45]
> 2015-09-10_06:41:22.92460   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>  ~[na:1.7.0_45]



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-10298) Replaced dead node stayed in gossip forever

2015-09-17 Thread Jim Witschey (JIRA) 

[ 
https://issues.apache.org/jira/browse/CASSANDRA-10298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14804670#comment-14804670
 ] 

Jim Witschey commented on CASSANDRA-10298:
--

[~dikanggu] Thanks for the information. I just noticed the patch you attached; 
sorry I missed it earlier. What branch of Cassandra does it target? I'm 
assuming cassandra-2.1, since that's the version you marked this ticket with.

[~iamaleksey] May I assign you this ticket, as you're the most recent person to 
work with the hinted handoff manager (at least on trunk)? Or, can you recommend 
someone else to take this ticket?

> Replaced dead node stayed in gossip forever
> ---
>
> Key: CASSANDRA-10298
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10298
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Dikang Gu
> Attachments: CASSANDRA-10298.patch
>
>
> The dead node stayed in the nodetool status,
> DN  10.210.165.55379.76 GB  256 ?   null
> And in the log, it throws NPE when trying to remove it.
> {code}
> 2015-09-10_06:41:22.92453 ERROR 06:41:22 Exception in thread 
> Thread[GossipStage:1,5,main]
> 2015-09-10_06:41:22.92454 java.lang.NullPointerException: null
> 2015-09-10_06:41:22.92455   at 
> org.apache.cassandra.utils.UUIDGen.decompose(UUIDGen.java:100) 
> 2015-09-10_06:41:22.92455   at 
> org.apache.cassandra.db.HintedHandOffManager.deleteHintsForEndpoint(HintedHandOffManager.java:201)
>  
> 2015-09-10_06:41:22.92455   at 
> org.apache.cassandra.service.StorageService.excise(StorageService.java:1886) 
> 2015-09-10_06:41:22.92455   at 
> org.apache.cassandra.service.StorageService.excise(StorageService.java:1902) 
> 2015-09-10_06:41:22.92456   at 
> org.apache.cassandra.service.StorageService.handleStateLeft(StorageService.java:1805)
> 2015-09-10_06:41:22.92457   at 
> org.apache.cassandra.service.StorageService.onChange(StorageService.java:1473)
>  
> 2015-09-10_06:41:22.92457   at 
> org.apache.cassandra.service.StorageService.onJoin(StorageService.java:2099) 
> 2015-09-10_06:41:22.92457   at 
> org.apache.cassandra.gms.Gossiper.handleMajorStateChange(Gossiper.java:1009) 
> 2015-09-10_06:41:22.92458   at 
> org.apache.cassandra.gms.Gossiper.applyStateLocally(Gossiper.java:1085) 
> 2015-09-10_06:41:22.92458   at 
> org.apache.cassandra.gms.GossipDigestAck2VerbHandler.doVerb(GossipDigestAck2VerbHandler.java:49)
>  
> 2015-09-10_06:41:22.92458   at 
> org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:62) 
> 2015-09-10_06:41:22.92459   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>  ~[na:1.7.0_45]
> 2015-09-10_06:41:22.92460   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>  ~[na:1.7.0_45]
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-10298) Replaced dead node stayed in gossip forever

2015-09-17 Thread Dikang Gu (JIRA) 

[ 
https://issues.apache.org/jira/browse/CASSANDRA-10298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14805065#comment-14805065
 ] 

Dikang Gu commented on CASSANDRA-10298:
---

[~mambocab] yeah, the patch is targeting 2.1, and it is easy to be ported to 
other version since it's a very simple fix.

I still need to test it, and probably [~iamaleksey] can take a look at it, see 
if it makes sense?

Is it OK that I assign this jiar to myself?

> Replaced dead node stayed in gossip forever
> ---
>
> Key: CASSANDRA-10298
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10298
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Dikang Gu
> Attachments: CASSANDRA-10298.patch
>
>
> The dead node stayed in the nodetool status,
> DN  10.210.165.55379.76 GB  256 ?   null
> And in the log, it throws NPE when trying to remove it.
> {code}
> 2015-09-10_06:41:22.92453 ERROR 06:41:22 Exception in thread 
> Thread[GossipStage:1,5,main]
> 2015-09-10_06:41:22.92454 java.lang.NullPointerException: null
> 2015-09-10_06:41:22.92455   at 
> org.apache.cassandra.utils.UUIDGen.decompose(UUIDGen.java:100) 
> 2015-09-10_06:41:22.92455   at 
> org.apache.cassandra.db.HintedHandOffManager.deleteHintsForEndpoint(HintedHandOffManager.java:201)
>  
> 2015-09-10_06:41:22.92455   at 
> org.apache.cassandra.service.StorageService.excise(StorageService.java:1886) 
> 2015-09-10_06:41:22.92455   at 
> org.apache.cassandra.service.StorageService.excise(StorageService.java:1902) 
> 2015-09-10_06:41:22.92456   at 
> org.apache.cassandra.service.StorageService.handleStateLeft(StorageService.java:1805)
> 2015-09-10_06:41:22.92457   at 
> org.apache.cassandra.service.StorageService.onChange(StorageService.java:1473)
>  
> 2015-09-10_06:41:22.92457   at 
> org.apache.cassandra.service.StorageService.onJoin(StorageService.java:2099) 
> 2015-09-10_06:41:22.92457   at 
> org.apache.cassandra.gms.Gossiper.handleMajorStateChange(Gossiper.java:1009) 
> 2015-09-10_06:41:22.92458   at 
> org.apache.cassandra.gms.Gossiper.applyStateLocally(Gossiper.java:1085) 
> 2015-09-10_06:41:22.92458   at 
> org.apache.cassandra.gms.GossipDigestAck2VerbHandler.doVerb(GossipDigestAck2VerbHandler.java:49)
>  
> 2015-09-10_06:41:22.92458   at 
> org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:62) 
> 2015-09-10_06:41:22.92459   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>  ~[na:1.7.0_45]
> 2015-09-10_06:41:22.92460   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>  ~[na:1.7.0_45]
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-10298) Replaced dead node stayed in gossip forever

2015-09-18 Thread Jim Witschey (JIRA) 

[ 
https://issues.apache.org/jira/browse/CASSANDRA-10298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14875932#comment-14875932
 ] 

Jim Witschey commented on CASSANDRA-10298:
--

[~dikanggu] I'll assign you for now, and we can evaluate later depending on how 
review goes.

[~iamaleksey] In light of that, may I mark you as a reviewer on this?

> Replaced dead node stayed in gossip forever
> ---
>
> Key: CASSANDRA-10298
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10298
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Dikang Gu
> Attachments: CASSANDRA-10298.patch
>
>
> The dead node stayed in the nodetool status,
> DN  10.210.165.55379.76 GB  256 ?   null
> And in the log, it throws NPE when trying to remove it.
> {code}
> 2015-09-10_06:41:22.92453 ERROR 06:41:22 Exception in thread 
> Thread[GossipStage:1,5,main]
> 2015-09-10_06:41:22.92454 java.lang.NullPointerException: null
> 2015-09-10_06:41:22.92455   at 
> org.apache.cassandra.utils.UUIDGen.decompose(UUIDGen.java:100) 
> 2015-09-10_06:41:22.92455   at 
> org.apache.cassandra.db.HintedHandOffManager.deleteHintsForEndpoint(HintedHandOffManager.java:201)
>  
> 2015-09-10_06:41:22.92455   at 
> org.apache.cassandra.service.StorageService.excise(StorageService.java:1886) 
> 2015-09-10_06:41:22.92455   at 
> org.apache.cassandra.service.StorageService.excise(StorageService.java:1902) 
> 2015-09-10_06:41:22.92456   at 
> org.apache.cassandra.service.StorageService.handleStateLeft(StorageService.java:1805)
> 2015-09-10_06:41:22.92457   at 
> org.apache.cassandra.service.StorageService.onChange(StorageService.java:1473)
>  
> 2015-09-10_06:41:22.92457   at 
> org.apache.cassandra.service.StorageService.onJoin(StorageService.java:2099) 
> 2015-09-10_06:41:22.92457   at 
> org.apache.cassandra.gms.Gossiper.handleMajorStateChange(Gossiper.java:1009) 
> 2015-09-10_06:41:22.92458   at 
> org.apache.cassandra.gms.Gossiper.applyStateLocally(Gossiper.java:1085) 
> 2015-09-10_06:41:22.92458   at 
> org.apache.cassandra.gms.GossipDigestAck2VerbHandler.doVerb(GossipDigestAck2VerbHandler.java:49)
>  
> 2015-09-10_06:41:22.92458   at 
> org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:62) 
> 2015-09-10_06:41:22.92459   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>  ~[na:1.7.0_45]
> 2015-09-10_06:41:22.92460   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>  ~[na:1.7.0_45]
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-10298) Replaced dead node stayed in gossip forever

2015-09-18 Thread Aleksey Yeschenko (JIRA) 

[ 
https://issues.apache.org/jira/browse/CASSANDRA-10298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14875935#comment-14875935
 ] 

Aleksey Yeschenko commented on CASSANDRA-10298:
---

bq. In light of that, may I mark you as a reviewer on this?

Done. Next time just go ahead - I'll reassign if necessary.

> Replaced dead node stayed in gossip forever
> ---
>
> Key: CASSANDRA-10298
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10298
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Dikang Gu
> Attachments: CASSANDRA-10298.patch
>
>
> The dead node stayed in the nodetool status,
> DN  10.210.165.55379.76 GB  256 ?   null
> And in the log, it throws NPE when trying to remove it.
> {code}
> 2015-09-10_06:41:22.92453 ERROR 06:41:22 Exception in thread 
> Thread[GossipStage:1,5,main]
> 2015-09-10_06:41:22.92454 java.lang.NullPointerException: null
> 2015-09-10_06:41:22.92455   at 
> org.apache.cassandra.utils.UUIDGen.decompose(UUIDGen.java:100) 
> 2015-09-10_06:41:22.92455   at 
> org.apache.cassandra.db.HintedHandOffManager.deleteHintsForEndpoint(HintedHandOffManager.java:201)
>  
> 2015-09-10_06:41:22.92455   at 
> org.apache.cassandra.service.StorageService.excise(StorageService.java:1886) 
> 2015-09-10_06:41:22.92455   at 
> org.apache.cassandra.service.StorageService.excise(StorageService.java:1902) 
> 2015-09-10_06:41:22.92456   at 
> org.apache.cassandra.service.StorageService.handleStateLeft(StorageService.java:1805)
> 2015-09-10_06:41:22.92457   at 
> org.apache.cassandra.service.StorageService.onChange(StorageService.java:1473)
>  
> 2015-09-10_06:41:22.92457   at 
> org.apache.cassandra.service.StorageService.onJoin(StorageService.java:2099) 
> 2015-09-10_06:41:22.92457   at 
> org.apache.cassandra.gms.Gossiper.handleMajorStateChange(Gossiper.java:1009) 
> 2015-09-10_06:41:22.92458   at 
> org.apache.cassandra.gms.Gossiper.applyStateLocally(Gossiper.java:1085) 
> 2015-09-10_06:41:22.92458   at 
> org.apache.cassandra.gms.GossipDigestAck2VerbHandler.doVerb(GossipDigestAck2VerbHandler.java:49)
>  
> 2015-09-10_06:41:22.92458   at 
> org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:62) 
> 2015-09-10_06:41:22.92459   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>  ~[na:1.7.0_45]
> 2015-09-10_06:41:22.92460   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>  ~[na:1.7.0_45]
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-10298) Replaced dead node stayed in gossip forever

2015-09-26 Thread Stefania (JIRA) 

[ 
https://issues.apache.org/jira/browse/CASSANDRA-10298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14909392#comment-14909392
 ] 

Stefania commented on CASSANDRA-10298:
--

It seems it's the same exception as in CASSANDRA-10231. As for the part of the 
patch that deals with tokens being null, for that there is a fix in 
CASSANDRA-10089.

> Replaced dead node stayed in gossip forever
> ---
>
> Key: CASSANDRA-10298
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10298
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Dikang Gu
>Assignee: Dikang Gu
> Fix For: 2.1.x
>
> Attachments: CASSANDRA-10298.patch
>
>
> The dead node stayed in the nodetool status,
> DN  10.210.165.55379.76 GB  256 ?   null
> And in the log, it throws NPE when trying to remove it.
> {code}
> 2015-09-10_06:41:22.92453 ERROR 06:41:22 Exception in thread 
> Thread[GossipStage:1,5,main]
> 2015-09-10_06:41:22.92454 java.lang.NullPointerException: null
> 2015-09-10_06:41:22.92455   at 
> org.apache.cassandra.utils.UUIDGen.decompose(UUIDGen.java:100) 
> 2015-09-10_06:41:22.92455   at 
> org.apache.cassandra.db.HintedHandOffManager.deleteHintsForEndpoint(HintedHandOffManager.java:201)
>  
> 2015-09-10_06:41:22.92455   at 
> org.apache.cassandra.service.StorageService.excise(StorageService.java:1886) 
> 2015-09-10_06:41:22.92455   at 
> org.apache.cassandra.service.StorageService.excise(StorageService.java:1902) 
> 2015-09-10_06:41:22.92456   at 
> org.apache.cassandra.service.StorageService.handleStateLeft(StorageService.java:1805)
> 2015-09-10_06:41:22.92457   at 
> org.apache.cassandra.service.StorageService.onChange(StorageService.java:1473)
>  
> 2015-09-10_06:41:22.92457   at 
> org.apache.cassandra.service.StorageService.onJoin(StorageService.java:2099) 
> 2015-09-10_06:41:22.92457   at 
> org.apache.cassandra.gms.Gossiper.handleMajorStateChange(Gossiper.java:1009) 
> 2015-09-10_06:41:22.92458   at 
> org.apache.cassandra.gms.Gossiper.applyStateLocally(Gossiper.java:1085) 
> 2015-09-10_06:41:22.92458   at 
> org.apache.cassandra.gms.GossipDigestAck2VerbHandler.doVerb(GossipDigestAck2VerbHandler.java:49)
>  
> 2015-09-10_06:41:22.92458   at 
> org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:62) 
> 2015-09-10_06:41:22.92459   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>  ~[na:1.7.0_45]
> 2015-09-10_06:41:22.92460   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>  ~[na:1.7.0_45]
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-10298) Replaced dead node stayed in gossip forever

2015-09-28 Thread Dikang Gu (JIRA) 

[ 
https://issues.apache.org/jira/browse/CASSANDRA-10298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14933609#comment-14933609
 ] 

Dikang Gu commented on CASSANDRA-10298:
---

[~Stefania], yeah, looks like the same issues, have you committed your patches 
to 2.1 branch?

> Replaced dead node stayed in gossip forever
> ---
>
> Key: CASSANDRA-10298
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10298
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Dikang Gu
>Assignee: Dikang Gu
> Fix For: 2.1.x
>
> Attachments: CASSANDRA-10298.patch
>
>
> The dead node stayed in the nodetool status,
> DN  10.210.165.55379.76 GB  256 ?   null
> And in the log, it throws NPE when trying to remove it.
> {code}
> 2015-09-10_06:41:22.92453 ERROR 06:41:22 Exception in thread 
> Thread[GossipStage:1,5,main]
> 2015-09-10_06:41:22.92454 java.lang.NullPointerException: null
> 2015-09-10_06:41:22.92455   at 
> org.apache.cassandra.utils.UUIDGen.decompose(UUIDGen.java:100) 
> 2015-09-10_06:41:22.92455   at 
> org.apache.cassandra.db.HintedHandOffManager.deleteHintsForEndpoint(HintedHandOffManager.java:201)
>  
> 2015-09-10_06:41:22.92455   at 
> org.apache.cassandra.service.StorageService.excise(StorageService.java:1886) 
> 2015-09-10_06:41:22.92455   at 
> org.apache.cassandra.service.StorageService.excise(StorageService.java:1902) 
> 2015-09-10_06:41:22.92456   at 
> org.apache.cassandra.service.StorageService.handleStateLeft(StorageService.java:1805)
> 2015-09-10_06:41:22.92457   at 
> org.apache.cassandra.service.StorageService.onChange(StorageService.java:1473)
>  
> 2015-09-10_06:41:22.92457   at 
> org.apache.cassandra.service.StorageService.onJoin(StorageService.java:2099) 
> 2015-09-10_06:41:22.92457   at 
> org.apache.cassandra.gms.Gossiper.handleMajorStateChange(Gossiper.java:1009) 
> 2015-09-10_06:41:22.92458   at 
> org.apache.cassandra.gms.Gossiper.applyStateLocally(Gossiper.java:1085) 
> 2015-09-10_06:41:22.92458   at 
> org.apache.cassandra.gms.GossipDigestAck2VerbHandler.doVerb(GossipDigestAck2VerbHandler.java:49)
>  
> 2015-09-10_06:41:22.92458   at 
> org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:62) 
> 2015-09-10_06:41:22.92459   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>  ~[na:1.7.0_45]
> 2015-09-10_06:41:22.92460   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>  ~[na:1.7.0_45]
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-10298) Replaced dead node stayed in gossip forever

2015-09-28 Thread Stefania (JIRA) 

[ 
https://issues.apache.org/jira/browse/CASSANDRA-10298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14934354#comment-14934354
 ] 

Stefania commented on CASSANDRA-10298:
--

No they are both still under test, besides we are focusing on the 3.0 branch 
for the tests. If you really need this in 2.1 then you can commit this patch.

> Replaced dead node stayed in gossip forever
> ---
>
> Key: CASSANDRA-10298
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10298
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Dikang Gu
>Assignee: Dikang Gu
> Fix For: 2.1.x
>
> Attachments: CASSANDRA-10298.patch
>
>
> The dead node stayed in the nodetool status,
> DN  10.210.165.55379.76 GB  256 ?   null
> And in the log, it throws NPE when trying to remove it.
> {code}
> 2015-09-10_06:41:22.92453 ERROR 06:41:22 Exception in thread 
> Thread[GossipStage:1,5,main]
> 2015-09-10_06:41:22.92454 java.lang.NullPointerException: null
> 2015-09-10_06:41:22.92455   at 
> org.apache.cassandra.utils.UUIDGen.decompose(UUIDGen.java:100) 
> 2015-09-10_06:41:22.92455   at 
> org.apache.cassandra.db.HintedHandOffManager.deleteHintsForEndpoint(HintedHandOffManager.java:201)
>  
> 2015-09-10_06:41:22.92455   at 
> org.apache.cassandra.service.StorageService.excise(StorageService.java:1886) 
> 2015-09-10_06:41:22.92455   at 
> org.apache.cassandra.service.StorageService.excise(StorageService.java:1902) 
> 2015-09-10_06:41:22.92456   at 
> org.apache.cassandra.service.StorageService.handleStateLeft(StorageService.java:1805)
> 2015-09-10_06:41:22.92457   at 
> org.apache.cassandra.service.StorageService.onChange(StorageService.java:1473)
>  
> 2015-09-10_06:41:22.92457   at 
> org.apache.cassandra.service.StorageService.onJoin(StorageService.java:2099) 
> 2015-09-10_06:41:22.92457   at 
> org.apache.cassandra.gms.Gossiper.handleMajorStateChange(Gossiper.java:1009) 
> 2015-09-10_06:41:22.92458   at 
> org.apache.cassandra.gms.Gossiper.applyStateLocally(Gossiper.java:1085) 
> 2015-09-10_06:41:22.92458   at 
> org.apache.cassandra.gms.GossipDigestAck2VerbHandler.doVerb(GossipDigestAck2VerbHandler.java:49)
>  
> 2015-09-10_06:41:22.92458   at 
> org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:62) 
> 2015-09-10_06:41:22.92459   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>  ~[na:1.7.0_45]
> 2015-09-10_06:41:22.92460   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>  ~[na:1.7.0_45]
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-10298) Replaced dead node stayed in gossip forever

2015-09-29 Thread Dikang Gu (JIRA) 

[ 
https://issues.apache.org/jira/browse/CASSANDRA-10298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14935515#comment-14935515
 ] 

Dikang Gu commented on CASSANDRA-10298:
---

[~Stefania], I see, yes, I'd like to check it in 2.1, how can I commit it?

> Replaced dead node stayed in gossip forever
> ---
>
> Key: CASSANDRA-10298
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10298
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Dikang Gu
>Assignee: Dikang Gu
> Fix For: 2.1.x
>
> Attachments: CASSANDRA-10298.patch
>
>
> The dead node stayed in the nodetool status,
> DN  10.210.165.55379.76 GB  256 ?   null
> And in the log, it throws NPE when trying to remove it.
> {code}
> 2015-09-10_06:41:22.92453 ERROR 06:41:22 Exception in thread 
> Thread[GossipStage:1,5,main]
> 2015-09-10_06:41:22.92454 java.lang.NullPointerException: null
> 2015-09-10_06:41:22.92455   at 
> org.apache.cassandra.utils.UUIDGen.decompose(UUIDGen.java:100) 
> 2015-09-10_06:41:22.92455   at 
> org.apache.cassandra.db.HintedHandOffManager.deleteHintsForEndpoint(HintedHandOffManager.java:201)
>  
> 2015-09-10_06:41:22.92455   at 
> org.apache.cassandra.service.StorageService.excise(StorageService.java:1886) 
> 2015-09-10_06:41:22.92455   at 
> org.apache.cassandra.service.StorageService.excise(StorageService.java:1902) 
> 2015-09-10_06:41:22.92456   at 
> org.apache.cassandra.service.StorageService.handleStateLeft(StorageService.java:1805)
> 2015-09-10_06:41:22.92457   at 
> org.apache.cassandra.service.StorageService.onChange(StorageService.java:1473)
>  
> 2015-09-10_06:41:22.92457   at 
> org.apache.cassandra.service.StorageService.onJoin(StorageService.java:2099) 
> 2015-09-10_06:41:22.92457   at 
> org.apache.cassandra.gms.Gossiper.handleMajorStateChange(Gossiper.java:1009) 
> 2015-09-10_06:41:22.92458   at 
> org.apache.cassandra.gms.Gossiper.applyStateLocally(Gossiper.java:1085) 
> 2015-09-10_06:41:22.92458   at 
> org.apache.cassandra.gms.GossipDigestAck2VerbHandler.doVerb(GossipDigestAck2VerbHandler.java:49)
>  
> 2015-09-10_06:41:22.92458   at 
> org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:62) 
> 2015-09-10_06:41:22.92459   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>  ~[na:1.7.0_45]
> 2015-09-10_06:41:22.92460   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>  ~[na:1.7.0_45]
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-10298) Replaced dead node stayed in gossip forever

2015-09-30 Thread Stefania (JIRA) 

[ 
https://issues.apache.org/jira/browse/CASSANDRA-10298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14939214#comment-14939214
 ] 

Stefania commented on CASSANDRA-10298:
--

Sorry I wasn't very clear: the patch needs to be reviewed, then a committer 
will be able to commit it.

However, I would like to reproduce this problem in a dtest first, could you 
give us the exact steps on how to reproduce it? Did you follow [these 
instructions|http://docs.datastax.com/en/cassandra/2.1/cassandra/operations/ops_replace_node_t.html]
 or deviated in any way?

In fact from a closer look, this is a slightly different problem than 
CASSANDRA-10231 because there we are processing new states ({{applyNewStates}}) 
whereas here we have received a major update ({{handleMajorStateChange}}).

> Replaced dead node stayed in gossip forever
> ---
>
> Key: CASSANDRA-10298
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10298
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Dikang Gu
>Assignee: Dikang Gu
> Fix For: 2.1.x
>
> Attachments: CASSANDRA-10298.patch
>
>
> The dead node stayed in the nodetool status,
> DN  10.210.165.55379.76 GB  256 ?   null
> And in the log, it throws NPE when trying to remove it.
> {code}
> 2015-09-10_06:41:22.92453 ERROR 06:41:22 Exception in thread 
> Thread[GossipStage:1,5,main]
> 2015-09-10_06:41:22.92454 java.lang.NullPointerException: null
> 2015-09-10_06:41:22.92455   at 
> org.apache.cassandra.utils.UUIDGen.decompose(UUIDGen.java:100) 
> 2015-09-10_06:41:22.92455   at 
> org.apache.cassandra.db.HintedHandOffManager.deleteHintsForEndpoint(HintedHandOffManager.java:201)
>  
> 2015-09-10_06:41:22.92455   at 
> org.apache.cassandra.service.StorageService.excise(StorageService.java:1886) 
> 2015-09-10_06:41:22.92455   at 
> org.apache.cassandra.service.StorageService.excise(StorageService.java:1902) 
> 2015-09-10_06:41:22.92456   at 
> org.apache.cassandra.service.StorageService.handleStateLeft(StorageService.java:1805)
> 2015-09-10_06:41:22.92457   at 
> org.apache.cassandra.service.StorageService.onChange(StorageService.java:1473)
>  
> 2015-09-10_06:41:22.92457   at 
> org.apache.cassandra.service.StorageService.onJoin(StorageService.java:2099) 
> 2015-09-10_06:41:22.92457   at 
> org.apache.cassandra.gms.Gossiper.handleMajorStateChange(Gossiper.java:1009) 
> 2015-09-10_06:41:22.92458   at 
> org.apache.cassandra.gms.Gossiper.applyStateLocally(Gossiper.java:1085) 
> 2015-09-10_06:41:22.92458   at 
> org.apache.cassandra.gms.GossipDigestAck2VerbHandler.doVerb(GossipDigestAck2VerbHandler.java:49)
>  
> 2015-09-10_06:41:22.92458   at 
> org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:62) 
> 2015-09-10_06:41:22.92459   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>  ~[na:1.7.0_45]
> 2015-09-10_06:41:22.92460   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>  ~[na:1.7.0_45]
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-10298) Replaced dead node stayed in gossip forever

2015-10-30 Thread Aleksey Yeschenko (JIRA) 

[ 
https://issues.apache.org/jira/browse/CASSANDRA-10298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14982521#comment-14982521
 ] 

Aleksey Yeschenko commented on CASSANDRA-10298:
---

[~Stefania] Reassigning reviewership to you as you see to be aware of the 
context.

> Replaced dead node stayed in gossip forever
> ---
>
> Key: CASSANDRA-10298
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10298
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Dikang Gu
>Assignee: Dikang Gu
> Fix For: 2.1.x
>
> Attachments: CASSANDRA-10298.patch
>
>
> The dead node stayed in the nodetool status,
> DN  10.210.165.55379.76 GB  256 ?   null
> And in the log, it throws NPE when trying to remove it.
> {code}
> 2015-09-10_06:41:22.92453 ERROR 06:41:22 Exception in thread 
> Thread[GossipStage:1,5,main]
> 2015-09-10_06:41:22.92454 java.lang.NullPointerException: null
> 2015-09-10_06:41:22.92455   at 
> org.apache.cassandra.utils.UUIDGen.decompose(UUIDGen.java:100) 
> 2015-09-10_06:41:22.92455   at 
> org.apache.cassandra.db.HintedHandOffManager.deleteHintsForEndpoint(HintedHandOffManager.java:201)
>  
> 2015-09-10_06:41:22.92455   at 
> org.apache.cassandra.service.StorageService.excise(StorageService.java:1886) 
> 2015-09-10_06:41:22.92455   at 
> org.apache.cassandra.service.StorageService.excise(StorageService.java:1902) 
> 2015-09-10_06:41:22.92456   at 
> org.apache.cassandra.service.StorageService.handleStateLeft(StorageService.java:1805)
> 2015-09-10_06:41:22.92457   at 
> org.apache.cassandra.service.StorageService.onChange(StorageService.java:1473)
>  
> 2015-09-10_06:41:22.92457   at 
> org.apache.cassandra.service.StorageService.onJoin(StorageService.java:2099) 
> 2015-09-10_06:41:22.92457   at 
> org.apache.cassandra.gms.Gossiper.handleMajorStateChange(Gossiper.java:1009) 
> 2015-09-10_06:41:22.92458   at 
> org.apache.cassandra.gms.Gossiper.applyStateLocally(Gossiper.java:1085) 
> 2015-09-10_06:41:22.92458   at 
> org.apache.cassandra.gms.GossipDigestAck2VerbHandler.doVerb(GossipDigestAck2VerbHandler.java:49)
>  
> 2015-09-10_06:41:22.92458   at 
> org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:62) 
> 2015-09-10_06:41:22.92459   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>  ~[na:1.7.0_45]
> 2015-09-10_06:41:22.92460   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>  ~[na:1.7.0_45]
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-10298) Replaced dead node stayed in gossip forever

2015-11-01 Thread Stefania (JIRA) 

[ 
https://issues.apache.org/jira/browse/CASSANDRA-10298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14984617#comment-14984617
 ] 

Stefania commented on CASSANDRA-10298:
--

+1, pending CI here:

http://cassci.datastax.com/view/Dev/view/stef1927/job/stef1927-10298-2.1-testall/
http://cassci.datastax.com/view/Dev/view/stef1927/job/stef1927-10298-2.1-dtest/

The code that catches the exception in {{handleStateLeft()}} will be improved 
by CASSANDRA-10089 where we will return an empty collection in case of no 
tokens therefore removing the need for a try/catch. We must also remember to 
check for {{!tokens.isEmpty()}} in {{excise()}} but we'll deal with this in 
10089.

A test to reproduce this problem would be helpful to ensure we don't break it 
again later on. [~dikanggu] are you able to reproduce this regularly and if so 
what are the exact steps?




> Replaced dead node stayed in gossip forever
> ---
>
> Key: CASSANDRA-10298
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10298
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Dikang Gu
>Assignee: Dikang Gu
> Fix For: 2.1.x
>
> Attachments: CASSANDRA-10298.patch
>
>
> The dead node stayed in the nodetool status,
> DN  10.210.165.55379.76 GB  256 ?   null
> And in the log, it throws NPE when trying to remove it.
> {code}
> 2015-09-10_06:41:22.92453 ERROR 06:41:22 Exception in thread 
> Thread[GossipStage:1,5,main]
> 2015-09-10_06:41:22.92454 java.lang.NullPointerException: null
> 2015-09-10_06:41:22.92455   at 
> org.apache.cassandra.utils.UUIDGen.decompose(UUIDGen.java:100) 
> 2015-09-10_06:41:22.92455   at 
> org.apache.cassandra.db.HintedHandOffManager.deleteHintsForEndpoint(HintedHandOffManager.java:201)
>  
> 2015-09-10_06:41:22.92455   at 
> org.apache.cassandra.service.StorageService.excise(StorageService.java:1886) 
> 2015-09-10_06:41:22.92455   at 
> org.apache.cassandra.service.StorageService.excise(StorageService.java:1902) 
> 2015-09-10_06:41:22.92456   at 
> org.apache.cassandra.service.StorageService.handleStateLeft(StorageService.java:1805)
> 2015-09-10_06:41:22.92457   at 
> org.apache.cassandra.service.StorageService.onChange(StorageService.java:1473)
>  
> 2015-09-10_06:41:22.92457   at 
> org.apache.cassandra.service.StorageService.onJoin(StorageService.java:2099) 
> 2015-09-10_06:41:22.92457   at 
> org.apache.cassandra.gms.Gossiper.handleMajorStateChange(Gossiper.java:1009) 
> 2015-09-10_06:41:22.92458   at 
> org.apache.cassandra.gms.Gossiper.applyStateLocally(Gossiper.java:1085) 
> 2015-09-10_06:41:22.92458   at 
> org.apache.cassandra.gms.GossipDigestAck2VerbHandler.doVerb(GossipDigestAck2VerbHandler.java:49)
>  
> 2015-09-10_06:41:22.92458   at 
> org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:62) 
> 2015-09-10_06:41:22.92459   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>  ~[na:1.7.0_45]
> 2015-09-10_06:41:22.92460   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>  ~[na:1.7.0_45]
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-10298) Replaced dead node stayed in gossip forever

2015-11-02 Thread Stefania (JIRA) 

[ 
https://issues.apache.org/jira/browse/CASSANDRA-10298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14984888#comment-14984888
 ] 

Stefania commented on CASSANDRA-10298:
--

CI is similar to the unpatched branch. Unfortunately at the moment we have many 
dtests failing on 2.1.

> Replaced dead node stayed in gossip forever
> ---
>
> Key: CASSANDRA-10298
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10298
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Dikang Gu
>Assignee: Dikang Gu
> Fix For: 2.1.x
>
> Attachments: CASSANDRA-10298.patch
>
>
> The dead node stayed in the nodetool status,
> DN  10.210.165.55379.76 GB  256 ?   null
> And in the log, it throws NPE when trying to remove it.
> {code}
> 2015-09-10_06:41:22.92453 ERROR 06:41:22 Exception in thread 
> Thread[GossipStage:1,5,main]
> 2015-09-10_06:41:22.92454 java.lang.NullPointerException: null
> 2015-09-10_06:41:22.92455   at 
> org.apache.cassandra.utils.UUIDGen.decompose(UUIDGen.java:100) 
> 2015-09-10_06:41:22.92455   at 
> org.apache.cassandra.db.HintedHandOffManager.deleteHintsForEndpoint(HintedHandOffManager.java:201)
>  
> 2015-09-10_06:41:22.92455   at 
> org.apache.cassandra.service.StorageService.excise(StorageService.java:1886) 
> 2015-09-10_06:41:22.92455   at 
> org.apache.cassandra.service.StorageService.excise(StorageService.java:1902) 
> 2015-09-10_06:41:22.92456   at 
> org.apache.cassandra.service.StorageService.handleStateLeft(StorageService.java:1805)
> 2015-09-10_06:41:22.92457   at 
> org.apache.cassandra.service.StorageService.onChange(StorageService.java:1473)
>  
> 2015-09-10_06:41:22.92457   at 
> org.apache.cassandra.service.StorageService.onJoin(StorageService.java:2099) 
> 2015-09-10_06:41:22.92457   at 
> org.apache.cassandra.gms.Gossiper.handleMajorStateChange(Gossiper.java:1009) 
> 2015-09-10_06:41:22.92458   at 
> org.apache.cassandra.gms.Gossiper.applyStateLocally(Gossiper.java:1085) 
> 2015-09-10_06:41:22.92458   at 
> org.apache.cassandra.gms.GossipDigestAck2VerbHandler.doVerb(GossipDigestAck2VerbHandler.java:49)
>  
> 2015-09-10_06:41:22.92458   at 
> org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:62) 
> 2015-09-10_06:41:22.92459   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>  ~[na:1.7.0_45]
> 2015-09-10_06:41:22.92460   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>  ~[na:1.7.0_45]
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-10298) Replaced dead node stayed in gossip forever

2015-11-04 Thread Aleksey Yeschenko (JIRA) 

[ 
https://issues.apache.org/jira/browse/CASSANDRA-10298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14989584#comment-14989584
 ] 

Aleksey Yeschenko commented on CASSANDRA-10298:
---

Committed as 
[b6015fa4883182531d66527780fce34c9b125e46|https://github.com/apache/cassandra/commit/b6015fa4883182531d66527780fce34c9b125e46]
 to 2.1, merged with 2.2, 3.0, and trunk.

[~Stefania] [~dikanggu] There was an issue with braces/newlines that went 
against code style, I fixed that on commit. I hope nothing else was overlooked 
though.

> Replaced dead node stayed in gossip forever
> ---
>
> Key: CASSANDRA-10298
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10298
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Dikang Gu
>Assignee: Dikang Gu
> Fix For: 2.1.x
>
> Attachments: CASSANDRA-10298.patch
>
>
> The dead node stayed in the nodetool status,
> DN  10.210.165.55379.76 GB  256 ?   null
> And in the log, it throws NPE when trying to remove it.
> {code}
> 2015-09-10_06:41:22.92453 ERROR 06:41:22 Exception in thread 
> Thread[GossipStage:1,5,main]
> 2015-09-10_06:41:22.92454 java.lang.NullPointerException: null
> 2015-09-10_06:41:22.92455   at 
> org.apache.cassandra.utils.UUIDGen.decompose(UUIDGen.java:100) 
> 2015-09-10_06:41:22.92455   at 
> org.apache.cassandra.db.HintedHandOffManager.deleteHintsForEndpoint(HintedHandOffManager.java:201)
>  
> 2015-09-10_06:41:22.92455   at 
> org.apache.cassandra.service.StorageService.excise(StorageService.java:1886) 
> 2015-09-10_06:41:22.92455   at 
> org.apache.cassandra.service.StorageService.excise(StorageService.java:1902) 
> 2015-09-10_06:41:22.92456   at 
> org.apache.cassandra.service.StorageService.handleStateLeft(StorageService.java:1805)
> 2015-09-10_06:41:22.92457   at 
> org.apache.cassandra.service.StorageService.onChange(StorageService.java:1473)
>  
> 2015-09-10_06:41:22.92457   at 
> org.apache.cassandra.service.StorageService.onJoin(StorageService.java:2099) 
> 2015-09-10_06:41:22.92457   at 
> org.apache.cassandra.gms.Gossiper.handleMajorStateChange(Gossiper.java:1009) 
> 2015-09-10_06:41:22.92458   at 
> org.apache.cassandra.gms.Gossiper.applyStateLocally(Gossiper.java:1085) 
> 2015-09-10_06:41:22.92458   at 
> org.apache.cassandra.gms.GossipDigestAck2VerbHandler.doVerb(GossipDigestAck2VerbHandler.java:49)
>  
> 2015-09-10_06:41:22.92458   at 
> org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:62) 
> 2015-09-10_06:41:22.92459   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>  ~[na:1.7.0_45]
> 2015-09-10_06:41:22.92460   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>  ~[na:1.7.0_45]
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-10298) Replaced dead node stayed in gossip forever

2015-11-04 Thread Dikang Gu (JIRA) 

[ 
https://issues.apache.org/jira/browse/CASSANDRA-10298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14989972#comment-14989972
 ] 

Dikang Gu commented on CASSANDRA-10298:
---

Thanks [~iamaleksey] and [~Stefania]!

> Replaced dead node stayed in gossip forever
> ---
>
> Key: CASSANDRA-10298
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10298
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Dikang Gu
>Assignee: Dikang Gu
> Fix For: 2.2.4, 2.1.12, 3.0.0
>
> Attachments: CASSANDRA-10298.patch
>
>
> The dead node stayed in the nodetool status,
> DN  10.210.165.55379.76 GB  256 ?   null
> And in the log, it throws NPE when trying to remove it.
> {code}
> 2015-09-10_06:41:22.92453 ERROR 06:41:22 Exception in thread 
> Thread[GossipStage:1,5,main]
> 2015-09-10_06:41:22.92454 java.lang.NullPointerException: null
> 2015-09-10_06:41:22.92455   at 
> org.apache.cassandra.utils.UUIDGen.decompose(UUIDGen.java:100) 
> 2015-09-10_06:41:22.92455   at 
> org.apache.cassandra.db.HintedHandOffManager.deleteHintsForEndpoint(HintedHandOffManager.java:201)
>  
> 2015-09-10_06:41:22.92455   at 
> org.apache.cassandra.service.StorageService.excise(StorageService.java:1886) 
> 2015-09-10_06:41:22.92455   at 
> org.apache.cassandra.service.StorageService.excise(StorageService.java:1902) 
> 2015-09-10_06:41:22.92456   at 
> org.apache.cassandra.service.StorageService.handleStateLeft(StorageService.java:1805)
> 2015-09-10_06:41:22.92457   at 
> org.apache.cassandra.service.StorageService.onChange(StorageService.java:1473)
>  
> 2015-09-10_06:41:22.92457   at 
> org.apache.cassandra.service.StorageService.onJoin(StorageService.java:2099) 
> 2015-09-10_06:41:22.92457   at 
> org.apache.cassandra.gms.Gossiper.handleMajorStateChange(Gossiper.java:1009) 
> 2015-09-10_06:41:22.92458   at 
> org.apache.cassandra.gms.Gossiper.applyStateLocally(Gossiper.java:1085) 
> 2015-09-10_06:41:22.92458   at 
> org.apache.cassandra.gms.GossipDigestAck2VerbHandler.doVerb(GossipDigestAck2VerbHandler.java:49)
>  
> 2015-09-10_06:41:22.92458   at 
> org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:62) 
> 2015-09-10_06:41:22.92459   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>  ~[na:1.7.0_45]
> 2015-09-10_06:41:22.92460   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>  ~[na:1.7.0_45]
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)