[ https://issues.apache.org/jira/browse/CASSANDRA-7307?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14018106#comment-14018106 ]
Brandon Williams commented on CASSANDRA-7307: --------------------------------------------- Going to 3s is still slightly dangerous for replace, taking ~54s in my test above. bq. I'm not sure it's possible to have it both ways without changing the design I agree with that, but I think we should err on the common side. bq. I think you'd need gossip arrivalwindows for every peer so that you could pass a "history" to new nodes to fix this. I think that goes against the FD's principle, since it'd be passing what is essentially invalid latency history, especially cross-dc. > New nodes mark dead nodes as up for 10 minutes > ---------------------------------------------- > > Key: CASSANDRA-7307 > URL: https://issues.apache.org/jira/browse/CASSANDRA-7307 > Project: Cassandra > Issue Type: Bug > Reporter: Richard Low > Assignee: Brandon Williams > Fix For: 1.2.17 > > > When doing a node replacement when other nodes are down we see the down nodes > marked as up for about 10 minutes. This means requests are routed to the dead > nodes causing timeouts. It also means replacing a node when multiple nodes > from a replica set is extremely difficult - the node usually tries to stream > from a dead node and the replacement fails. > This isn't limited to host replacement. I did a simple test: > 1. Create a 2 node cluster > 2. Kill node 2 > 3. Start a 3rd node with a unique token (I used auto_bootstrap=false but I > don't think this is significant) > The 3rd node lists node 2 (127.0.0.2) as up for almost 10 minutes: > {code} > INFO [main] 2014-05-27 14:28:24,753 CassandraDaemon.java (line 119) Logging > initialized > INFO [GossipStage:1] 2014-05-27 14:28:31,492 Gossiper.java (line 843) Node > /127.0.0.2 is now part of the cluster > INFO [GossipStage:1] 2014-05-27 14:28:31,495 Gossiper.java (line 809) > InetAddress /127.0.0.2 is now UP > INFO [GossipTasks:1] 2014-05-27 14:37:44,526 Gossiper.java (line 823) > InetAddress /127.0.0.2 is now DOWN > {code} > I reproduced on 1.2.15 and 1.2.16. -- This message was sent by Atlassian JIRA (v6.2#6252)