Potential bootstrap failure bugs

Hi, I have tested Cassandra 3.0.10 and 3.0.17 and I have found three potential bugs there. The test steps with CCM are as follows: 1) Start 3 node cluster 2) Create enough data with cassandra-stress 3) Kill node3 4) Create + Start node4 with replace_address=node3 5) Wait for streaming to start

bootstrap failure and strange gossiper state

I am also experiencing issues bootstrapping new nodes in my 2.0.10 Cassandra cluster. The first attempt to bootstrap ALWAYS fails, followed by a second bootstrap attempt that ALWAYS succeeds. The first attempt at bootstrapping fails with: INFO [main] 2015-03-15 02:41:02,550 StorageService.java

Re: Bootstrap failure on C* 1.2.13

Hello, After about 3 months I was able to solve this issue, which happened again after another node died. The problem is the datastax 1.2 node replacement docs [1] said that "This procedure applies to clusters using vnodes. If not using vnodes, use the instructions in the Cassandra 1.1 documentat

Re: Bootstrap failure on C* 1.2.13

Hi Rob, I don't understand how setting those "initial_token" might solve this issue. Even more since we cannot set them before bootstrapping... Plus, once those tokens set, we would have to modify them after any new bootstrap / decommission. Which would also imply to run a rolling restart for the

Re: Bootstrap failure on C* 1.2.13

On Fri, Feb 14, 2014 at 10:08 AM, Paulo Ricardo Motta Gomes < paulo.mo...@chaordicsystems.com> wrote: > But in our case, our cluster was not using VNodes, so this workaround will > probably not work with VNodes, since you cannot specify the 256 tokens from > the old node. > Sure you can, in a com

Re: Bootstrap failure on C* 1.2.13

Hello Alain, I solved this with a brute force solution, but didn't understand exactly what happened behind the scenes. What I did was: a) removed the failed node from the ring with the unsafeAssassinate JMX option. b) this caused requests to that node to be routed to the following node which didn

Re: Bootstrap failure on C* 1.2.13

Hi Paulo, Did you find out how to fix this issue ? I am experimenting the exact same issue after trying to help you on this exact subject a few days ago :). Config : 32 C*1.2.11 nodes, Vnodes enabled, RF=3, 1 DC, On AWS EC2 m1.xlarge. We added a few nodes (4) and it seems that this occurs on one

Re: Bootstrap failure on C* 1.2.13

On Fri, Feb 7, 2014 at 4:41 AM, Alain RODRIGUEZ wrote: > From changelog : > > 1.2.15 > * Move handling of migration event source to solve bootstrap race > (CASSANDRA-6648) > > Maybe should you give this new version a try, if you suspect your issue to be > related to CASSANDRA-6648. > > 6648 ap

Re: Bootstrap failure on C* 1.2.13

>From changelog : 1.2.15 * Move handling of migration event source to solve bootstrap race (CASSANDRA-6648) Maybe should you give this new version a try, if you suspect your issue to be related to CASSANDRA-6648. Hope this will solve your issue. 2014-02-06 16:48 GMT+01:00 Paulo Ricardo Motta

Bootstrap failure on C* 1.2.13

Hello, One the nodes of our cluster failed and I performed the "replace a dead node procedure" described in [1]. After about 5 or 6 hours of streaming during bootstrap, the node fails with the exception "Unable to fetch range for keyspace foobar from any hosts." [2]. I haven't found any thread or

Re: Bootstrap failure

assandra.apache.org>> Cc: Don Jackson mailto:djack...@nanigans.com>>, Dave Carroll mailto:dcarr...@nanigans.com>> Subject: Bootstrap failure Hi all, We have been struggling with the inability to bootstrap nodes into our 1.2.13 environment with Vnodes using centos 6.4 with Java

Bootstrap failure

Hi all, We have been struggling with the inability to bootstrap nodes into our 1.2.13 environment with Vnodes using centos 6.4 with Java 7. We have an 8 node cluster (32 GB RAM, dual hex core, SSDs, 8 GB heap with 1200 MB eden space, RF3) with around 1 TB per node using murmur3. When we