Chris Burroughs created CASSANDRA-5915:
------------------------------------------
Summary: node flapping prevents replace_node from succeeding
consistently
Key: CASSANDRA-5915
URL: https://issues.apache.org/jira/browse/CASSANDRA-5915
Project: Cassandra
Issue Type: Bug
Components: Core
Environment: 1.2.8
Reporter: Chris Burroughs
A node was down for a week or two due to hardware disk failure. I tried to use
replace_node to bring up a new node on the same physical host with the same
IPs. (rbranson suspected that using the same IP may be more issue prone.) This
failed due to "unable to find sufficient sources for streaming range" See
CASSANDRA-5913 for a problem with how the failure was handled by gossip.
All of the other nodes should have been up the entire time, but when this node
came up it saw nodes flap up and down for quiet some time. I was eventually
able to get replace_token to work by adding a 60 (!) second sleep to
StorageService:bootstrap. I don't know if the right path is "why are things
flapping so much" or "bootstrap should wait until things look stable".
A few notes about the cluster:
* 2 dc cluster (about 20 each), using GossipingPropertyFileSnitch
* multi-dc no vpn setup:
http://mail-archives.apache.org/mod_mbox/cassandra-user/201306.mbox/%3c51bf5c79.7020...@gmail.com%3E
Startup log from the successful (with sleep) replace_node attached.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
