Re: Thousands of SSTables generated in only one node

[Surbhi Gupta]

We have seen the issue while using LCS that there were around 100K stables
got generated and compactions were not able to catch up and node became
unresponsive. The reason for that was one of the stable got corrupted and
compaction was kind of hanging on that sstable and further sstables were
flushing . After running nodetool scrub all went well.
So corruption might also be the reason. After running nodetool scrub you
will see message in system.log if any of the sstable is corrupted and
nodetool scrub will also clean the corruption.

On 25 October 2016 at 11:30, Edward Capriolo  wrote:

> I have not read the entire thread so sorry if this is already mentioned.
> You should review your logs, a potential problem could be a corrupted
> sstable.
>
> In a situation like this you will notice that the system is repeatedly
> trying to compact a given sstable. The compaction fails and based on the
> heuristics it may successfully compact some other files, but ultimately
> each time it attempts to do a compaction involving this sstable the process
> fails and the number of files keeps growing.
>
> Good luck,
> Edward
>
> On Tue, Oct 25, 2016 at 10:31 AM, DuyHai Doan 
> wrote:
>
>> what are your disk hardware specs ?
>>
>> On Tue, Oct 25, 2016 at 8:47 AM, Lahiru Gamathige 
>> wrote:
>>
>>> Hi Users,
>>>
>>> I have a single server code deployed with multiple environments
>>> (staging, dev etc) but they all use a single Cassandra cluster but
>>> keyspaces are prefixed with the environment name, so each server has its
>>> own keyspace to store data. I am using Cassandra 2.1.0 and using it to
>>> store timeseries data.
>>>
>>> I see thousands of SSTables in only one node for one environment and
>>> that node is running out of memory because of this (I am guessing thats the
>>> cause because I see lots of logs trying to compact that data). All the
>>> other nodes which use other environments too are just works fine but this
>>> not with one environment keeps having this issue.
>>> Given that explanation I have two main questions.
>>>
>>> Anyone of you had the similar issue ? If so how did you solve it.
>>>
>>> If I want to clean only this keyspace from the full cluster what are the
>>> steps I should be doing ?
>>>
>>> Do you think if I shut down the cluster and delete the folder for the
>>> keyspace in all the nodes and restart the cluster would do the job ? Are
>>> there any other steps I need to follow ?
>>> (Reason I ask is if I just truncate from cql still data will be there
>>> and there's seriously something wrong in that table and I'm not sure it
>>> will ever get cleaned up)
>>>
>>> Thanks
>>> Lahiru
>>>
>>
>>
>

Re: Thousands of SSTables generated in only one node

[Edward Capriolo]

I have not read the entire thread so sorry if this is already mentioned.
You should review your logs, a potential problem could be a corrupted
sstable.

In a situation like this you will notice that the system is repeatedly
trying to compact a given sstable. The compaction fails and based on the
heuristics it may successfully compact some other files, but ultimately
each time it attempts to do a compaction involving this sstable the process
fails and the number of files keeps growing.

Good luck,
Edward

On Tue, Oct 25, 2016 at 10:31 AM, DuyHai Doan  wrote:

> what are your disk hardware specs ?
>
> On Tue, Oct 25, 2016 at 8:47 AM, Lahiru Gamathige 
> wrote:
>
>> Hi Users,
>>
>> I have a single server code deployed with multiple environments (staging,
>> dev etc) but they all use a single Cassandra cluster but keyspaces are
>> prefixed with the environment name, so each server has its own keyspace to
>> store data. I am using Cassandra 2.1.0 and using it to store timeseries
>> data.
>>
>> I see thousands of SSTables in only one node for one environment and that
>> node is running out of memory because of this (I am guessing thats the
>> cause because I see lots of logs trying to compact that data). All the
>> other nodes which use other environments too are just works fine but this
>> not with one environment keeps having this issue.
>> Given that explanation I have two main questions.
>>
>> Anyone of you had the similar issue ? If so how did you solve it.
>>
>> If I want to clean only this keyspace from the full cluster what are the
>> steps I should be doing ?
>>
>> Do you think if I shut down the cluster and delete the folder for the
>> keyspace in all the nodes and restart the cluster would do the job ? Are
>> there any other steps I need to follow ?
>> (Reason I ask is if I just truncate from cql still data will be there and
>> there's seriously something wrong in that table and I'm not sure it will
>> ever get cleaned up)
>>
>> Thanks
>> Lahiru
>>
>
>

Re: Thousands of SSTables generated in only one node

[DuyHai Doan]

what are your disk hardware specs ?

On Tue, Oct 25, 2016 at 8:47 AM, Lahiru Gamathige 
wrote:

> Hi Users,
>
> I have a single server code deployed with multiple environments (staging,
> dev etc) but they all use a single Cassandra cluster but keyspaces are
> prefixed with the environment name, so each server has its own keyspace to
> store data. I am using Cassandra 2.1.0 and using it to store timeseries
> data.
>
> I see thousands of SSTables in only one node for one environment and that
> node is running out of memory because of this (I am guessing thats the
> cause because I see lots of logs trying to compact that data). All the
> other nodes which use other environments too are just works fine but this
> not with one environment keeps having this issue.
> Given that explanation I have two main questions.
>
> Anyone of you had the similar issue ? If so how did you solve it.
>
> If I want to clean only this keyspace from the full cluster what are the
> steps I should be doing ?
>
> Do you think if I shut down the cluster and delete the folder for the
> keyspace in all the nodes and restart the cluster would do the job ? Are
> there any other steps I need to follow ?
> (Reason I ask is if I just truncate from cql still data will be there and
> there's seriously something wrong in that table and I'm not sure it will
> ever get cleaned up)
>
> Thanks
> Lahiru
>

Re: Thousands of SSTables generated in only one node

[kurt Greaves]

+1 definitely upgrade to 2.1.16. You shouldn't see any compatibility issues
client side when upgrading from 2.1.0. If scrub removed 500 SSTables that's
quite worrying. If the mass SSTables are causing issues you can disconnect
the node from the cluster using:
nodetool disablegossip && nodetool disablebinary && nodetool disablethrift
This will give it a chance to compact SSTables while not impacting read
performance. To be on the safe side you shouldn't have it disconnected for
longer than 3 hours (default hinted handoff window). You can re-enable it
with:
nodetool enablegossip && nodetool enablebinary && nodetool enablethrift
​

Re: Thousands of SSTables generated in only one node

[Jan Kesten]

Hi Lahiru,

2.1.0 is also quite old (Sep 2014) - and just from my memory I 
remembered that there was an issue whe had with cold_reads_to_omit:


http://grokbase.com/t/cassandra/user/1523sm4y0r/how-to-deal-with-too-many-sstables
https://www.mail-archive.com/search?l=user@cassandra.apache.org=subject:%22Re%3A+Compaction+failing+to+trigger%22=newest=1

That's just a random google hits but maybe that also helps.

I ended up with a few thousand sstables smaller than 1MB in size. 
However I would suggest upgrading to a newer version of cassandra first 
before diving too deep into this - maybe 2.1.16 or 2.2.8 - as chances 
are really good your problems will be gone after that.


Regards.
Jan

Re: Thousands of SSTables generated in only one node

[Lahiru Gamathige]

Hi Jan,

Thanks for the response. My SSTables are < 3MB and I have 3500+ SSTables in
the folder. When you say if they are small do you think my file sizes are
small ? I ran the nodetool compact nothing happened, then I ran nodetool
scrub it removed 500 SSTables then it stopped.

Thanks for that tip about the heap usage, it looks like a legit fact I
should be checking out.

Regards
Lahiru

On Mon, Oct 24, 2016 at 11:58 PM, Jan Kesten  wrote:

> Hi Lahiru,
>
> maybe your node was running out of memory before. I saw this behaviour if
> available heap is low forcing to flush out memtables to sstables quite
> often.
>
> If this is that what is hitting you, you should see that the sstables are
> really small.
>
> To cleanup, nodetool compact would do the job - but if you do not need
> data from one of the keyspaces at all just drop and recreate it (but look
> into your data directory if there are snapshots left). Prevent this in
> future have a close look at heap consumption and maybe give it more memory.
>
> HTH,
> Jan
>

Re: Thousands of SSTables generated in only one node

[Jan Kesten]

Hi Lahiru,

maybe your node was running out of memory before. I saw this behaviour 
if available heap is low forcing to flush out memtables to sstables 
quite often.


If this is that what is hitting you, you should see that the sstables 
are really small.


To cleanup, nodetool compact would do the job - but if you do not need 
data from one of the keyspaces at all just drop and recreate it (but 
look into your data directory if there are snapshots left). Prevent this 
in future have a close look at heap consumption and maybe give it more 
memory.


HTH,
Jan