Hello !
I am having a recurrent problem when trying to bootstrap a few new nodes.
Some general info :
- I am running cassandra 3.0.17
- We have about 30 nodes in our cluster
- All healthy nodes have between 60% to 90% used disk space on
/var/lib/cassandra
So I create a new node and
: Bootstrap keeps failing
Hello !
I am having a recurrent problem when trying to bootstrap a few new nodes.
Some general info :
* I am running cassandra 3.0.17
* We have about 30 nodes in our cluster
* All healthy nodes have between 60% to 90% used disk space on
/var
Would it be possible for you to take a thread dump & logs and share them?
Dinesh
On Wednesday, February 6, 2019, 10:09:11 AM PST, Léo FERLIN SUTTON
wrote:
Hello !
I am having a recurrent problem when trying to bootstrap a few new nodes.
Some general info :
- I am running cassand
: Léo FERLIN SUTTON [mailto:lfer...@mailjet.com.INVALID]
Sent: Thursday, February 07, 2019 9:16 AM
To: user@cassandra.apache.org
Subject: Re: [EXTERNAL] Re: Bootstrap keeps failing
Thank you for the recommendation.
We are already using datastax's recommended settings for tcp_keep
unch `nodetool bootstrap resume` on the instance.
Thank you for you interest in our issue !
Regards,
Leo
>
>
> *From:* Léo FERLIN SUTTON [mailto:lfer...@mailjet.com.INVALID]
> *Sent:* Thursday, February 07, 2019 9:16 AM
> *To:* user@cassandra.apache.org
> *Subject:* Re: [
s not we launch `nodetool bootstrap resume` on the instance.
Thank you for you interest in our issue !
Regards,
Leo
From: Léo FERLIN SUTTON [mailto:lfer...@mailjet.com.INVALID]
Sent: Thursday, February 07, 2019 9:16 AM
To: user@cassandra.apache.org
Subject: Re: [EXTERNAL] Re: Bootstra
.
>6. We start the cassandra.
>
> If after 3 days we see that the node hasn't joined the cluster we check
> the `nodetool netstats` command to see if the node is still streaming data.
> If it is not we launch `nodetool bootstrap resume` on the instance.
>
>
>
>
s and it looks like we
> have a few very big nodes instead of a lot of smaller ones.
> We are currently trying to add nodes, and are hoping to eventually
> transition to a "lot of small nodes" model and be able to add nodes a lot
> faster.
>
> Thank you again for your interest,
>
@cassandra.apache.org
Subject: Re: Bootstrap keeps failing
Hello again !
I have managed to free a lot of disk space and now most nodes hover between 50%
and 80%.
I am still getting bootstrapping failures :(
Here I have some logs :
2019-02-14T15:23:05+00:00 cass02-0001.c.company.internal user err
ology changes to cleanup. `nodetool cleanup` did miracles.
Regards,
Leo
>
> *From:* Léo FERLIN SUTTON [mailto:lfer...@mailjet.com.INVALID]
> *Sent:* Thursday, February 14, 2019 7:54 AM
> *To:* user@cassandra.apache.org
> *Subject:* Re: Bootstrap keeps failing
>
>
>
> Hello a
old snap shots that you can delete. And so on.
>>
>> Unfortunately no. We take a daily snapshot that we backup then drop.
>>
>>
>>
>> You have to make more room on the existing nodes.
>>
>>
>>
>> I am trying to run `nodetool cleanup` on our most &
gt;
>> Regards,
>>
>> Leo
>>
>>>
>>> *From:* Léo FERLIN SUTTON [mailto:lfer...@mailjet.com.INVALID]
>>> *Sent:* Thursday, February 14, 2019 7:54 AM
>>> *To:* user@cassandra.apache.org
>>> *Subject:* Re: Bootstrap keeps failing
>>>
keeps failing
Hello !
Thank you for your answers.
So I have tried, multiple times, to start bootstrapping from scratch. I often
have the same problem (on other nodes as well) but sometimes it works and I can
move on to another node.
I have joined a jstack dump and some logs.
Our node was shut
ess to the intermediate network equipment.
>
>
>
> Sean Durity
>
> *From:* Léo FERLIN SUTTON
> *Sent:* Thursday, February 07, 2019 10:26 AM
> *To:* user@cassandra.apache.org; dinesh.jo...@yahoo.com
> *Subject:* [EXTERNAL] Re: Bootstrap keeps failing
>
>
>
> H
14 matches
Mail list logo
