Re: Stress tool command

[Jeff Jirsa]

You can create the schema in advance with custom table options and stress will 
happily use it as-is


-- 
Jeff Jirsa


> On Sep 4, 2017, at 10:25 AM, Akshit Jain  wrote:
> 
> Hi,
> Is there any way to set the gc_grace_seconds parameter in the stress tool 
> command?
> 
> Regards
> 
> 

Stress tool command

[Akshit Jain]

Hi,
Is there any way to set the *gc_grace_seconds* parameter in the stress tool
command?

Regards

Re: Cassandra snapshot restore with VNODES missing some data

[Jai Bheemsen Rao Dhanwada]

Hello Kurt,

Thanks for the help :)

On Fri, Sep 1, 2017 at 1:12 PM, Jai Bheemsen Rao Dhanwada <
jaibheem...@gmail.com> wrote:

> yes looks like I am missing that.
>
> Let me test on one node and try a full cluster restore.
>
> will update here once I complete my test
>
> On Fri, Sep 1, 2017 at 5:01 AM, kurt greaves  wrote:
>
>> is num_tokens also set to 256?
>>
>
>

Re: old big tombstone data file occupy much disk space

[Shalom Sagges]

Thanks! :-)



On Mon, Sep 4, 2017 at 2:56 PM, Nicolas Guyomar 
wrote:

> Wrong copy/paste !
>
> Looking at the code, it should do nothing :
>
>  // look up the sstables now that we're on the compaction executor, so we
> don't try to re-compact
>  // something that was already being compacted earlier.
>
> On 4 September 2017 at 13:54, Nicolas Guyomar 
> wrote:
>
>> You'll get the WARN "Will not compact {}: it is not an active sstable"
>>  :)
>>
>> On 4 September 2017 at 12:07, Shalom Sagges 
>> wrote:
>>
>>> By the way, does anyone know what happens if I run a user defined
>>> compaction on an sstable that's already in compaction?
>>>
>>>
>>>
>>>
>>>
>>>
>>> On Sun, Sep 3, 2017 at 2:55 PM, Shalom Sagges 
>>> wrote:
>>>
 Try this blog by The Last Pickle:

 http://thelastpickle.com/blog/2016/10/18/user-defined-compaction.html






 Shalom Sagges
 DBA
 
   We
 Create Meaningful Connections



 On Sat, Sep 2, 2017 at 8:34 PM, Jeff Jirsa  wrote:

> If you're on 3.0 (3.0.6 or 3.0.8 or newer I don't remember which),
> TWCS was designed for ttl-only time series use cases
>
> Alternatively, if you have IO to spare, you may find LCS works as well
> (it'll cause quite a bit more compaction, but a much higher chance to
> compact away tombstones)
>
> There are also tombstone focused sub properties to more aggressively
> compact sstables that have a lot of tombstones - check the docs for
> "unchecked tombstone compaction" and "tombstone threshold" - enabling 
> those
> will enable more aggressive automatic single-sstable compactions
>
> --
> Jeff Jirsa
>
>
> On Sep 2, 2017, at 7:10 AM, qf zhou  wrote:
>
>
> Yes, your are right. I am using STCS compaction strategy with some
> kind of timeseries model. Too much disk space has been occupied.
>
> What should I  do to stop  the  disk full ?
>
>  I only want to keep 100 days data most recently,  so I set 
> default_time_to_live
> = 864(100 days ).
>
> I know I need to do something to stop the disk space cost, but I
> really don’t know how to do it.
>
>
> Here is the strategy of the big data table :
>
> AND compaction = {'class': 'org.apache.cassandra.db.compa
> ction.SizeTieredCompactionStrategy', 'max_threshold': '32',
> 'min_threshold': '12', 'tombstone_threshold': '0.1',
> 'unchecked_tombstone_compaction': 'true'}
> AND compression = {'chunk_length_in_kb': '64', 'class': '
> org.apache.cassandra.io.compress.LZ4Compressor'}
> AND crc_check_chance = 1.0
> AND dclocal_read_repair_chance = 0.1
> AND default_time_to_live = 864
> AND gc_grace_seconds = 432000
>
>
>
> 在 2017年9月2日，下午7:34，Nicolas Guyomar  写道：
>
> your are using STCS compaction strategy with some kind of timeseries
> model, and you are going to end up with yor disk full!
>
>
>

>>>
>>> This message may contain confidential and/or privileged information.
>>> If you are not the addressee or authorized to receive this on behalf of
>>> the addressee you must not use, copy, disclose or take action based on this
>>> message or any information herein.
>>> If you have received this message in error, please advise the sender
>>> immediately by reply email and delete this message. Thank you.
>>>
>>
>>
>

-- 
This message may contain confidential and/or privileged information. 
If you are not the addressee or authorized to receive this on behalf of the 
addressee you must not use, copy, disclose or take action based on this 
message or any information herein. 
If you have received this message in error, please advise the sender 
immediately by reply email and delete this message. Thank you.

Re: timeouts on counter tables

[Rudi Bruchez]

I'm going to try different options. Do any of you have some experience 
with tweaking one of those conf parameters to improve read throughput, 
especially in case of counter tables ?



1/ using SSD :
trickle_fsync: true
trickle_fsync_interval_in_kb: 1024

2/ concurrent_compactors to the number of cores.

3/ concurrent_counter_writes

4/ Row Cache vs Chunk Cache

5/ change the compaction method to leveled, specifically when using 
counter columns ??


thanks !

On 3 September 2017 at 20:25, Rudi Bruchez > wrote:


Le 30/08/2017 à 05:33, Erick Ramirez a écrit :

Is it possible at all that you may have a data hotspot if it's
not hardware-related?



It does not seem so, The partition key seems well distributed and
the queries update different keys.

We have dropped counter_mutation messages in the log :

COUNTER_MUTATION messages were dropped in last 5000 ms: 0
internal and 2 cross node. Mean internal dropped latency: 0 ms
and Mean cross-node dropped latency: 5960 ms

Pool NameActive   Pending Completed  
Blocked  All Time Blocked

ReadStage32 5037481787 0 0
CounterMutationStage 32   221 5722101
0 0


The load could be too high ?

Thanks






<>

smime.p7s
Description: Signature cryptographique S/MIME

Re: timeouts on counter tables

[Rudi Bruchez]

It can happen on any of the nodes. We can have a large number of pending 
on ReadStage and CounterMutationStage. We'll try to increase 
concurrent_counter_writes to see how it changes things


Likely. I believe counter mutations are a tad more expensive than a 
normal mutation. If you're doing a lot of counter updates that 
probably doesn't help. Regardless, high amounts of pending 
reads/mutations is generally not good and indicates the node being 
overloaded. Are you just seeing this on the 1 node with IO issues or 
do other nodes have this problem as well?


On 3 September 2017 at 20:25, Rudi Bruchez > wrote:


Le 30/08/2017 à 05:33, Erick Ramirez a écrit :

Is it possible at all that you may have a data hotspot if it's
not hardware-related?



It does not seem so, The partition key seems well distributed and
the queries update different keys.

We have dropped counter_mutation messages in the log :

COUNTER_MUTATION messages were dropped in last 5000 ms: 0 internal
and 2 cross node. Mean internal dropped latency: 0 ms and Mean
cross-node dropped latency: 5960 ms

Pool NameActive   Pending Completed   Blocked 
All Time Blocked

ReadStage32   503 7481787 0 0
CounterMutationStage 32   221 5722101
0 0


The load could be too high ?

Thanks




<>

smime.p7s
Description: Signature cryptographique S/MIME

Re: timeouts on counter tables

[kurt greaves]

Likely. I believe counter mutations are a tad more expensive than a normal
mutation. If you're doing a lot of counter updates that probably doesn't
help. Regardless, high amounts of pending reads/mutations is generally not
good and indicates the node being overloaded. Are you just seeing this on
the 1 node with IO issues or do other nodes have this problem as well?

On 3 September 2017 at 20:25, Rudi Bruchez  wrote:

> Le 30/08/2017 à 05:33, Erick Ramirez a écrit :
>
> Is it possible at all that you may have a data hotspot if it's not
> hardware-related?
>
>
> It does not seem so, The partition key seems well distributed and the
> queries update different keys.
>
> We have dropped counter_mutation messages in the log :
>
> COUNTER_MUTATION messages were dropped in last 5000 ms: 0 internal and 2
> cross node. Mean internal dropped latency: 0 ms and Mean cross-node dropped
> latency: 5960 ms
>
> Pool NameActive   Pending  Completed   Blocked
> All Time Blocked
> ReadStage32   5037481787
> 0 0
> CounterMutationStage 32   2215722101
> 0 0
>
> The load could be too high ?
>
> Thanks
>

Re: Test repair command

[kurt greaves]

Try checking the Percent Repaired reported in nodetool cfstats​

Re: old big tombstone data file occupy much disk space

[Nicolas Guyomar]

Wrong copy/paste !

Looking at the code, it should do nothing :

 // look up the sstables now that we're on the compaction executor, so we
don't try to re-compact
 // something that was already being compacted earlier.

On 4 September 2017 at 13:54, Nicolas Guyomar 
wrote:

> You'll get the WARN "Will not compact {}: it is not an active sstable"  :)
>
> On 4 September 2017 at 12:07, Shalom Sagges 
> wrote:
>
>> By the way, does anyone know what happens if I run a user defined
>> compaction on an sstable that's already in compaction?
>>
>>
>>
>>
>>
>>
>> On Sun, Sep 3, 2017 at 2:55 PM, Shalom Sagges 
>> wrote:
>>
>>> Try this blog by The Last Pickle:
>>>
>>> http://thelastpickle.com/blog/2016/10/18/user-defined-compaction.html
>>>
>>>
>>>
>>>
>>>
>>>
>>> Shalom Sagges
>>> DBA
>>>  
>>>  We Create Meaningful Connections
>>>
>>>
>>>
>>> On Sat, Sep 2, 2017 at 8:34 PM, Jeff Jirsa  wrote:
>>>
 If you're on 3.0 (3.0.6 or 3.0.8 or newer I don't remember which), TWCS
 was designed for ttl-only time series use cases

 Alternatively, if you have IO to spare, you may find LCS works as well
 (it'll cause quite a bit more compaction, but a much higher chance to
 compact away tombstones)

 There are also tombstone focused sub properties to more aggressively
 compact sstables that have a lot of tombstones - check the docs for
 "unchecked tombstone compaction" and "tombstone threshold" - enabling those
 will enable more aggressive automatic single-sstable compactions

 --
 Jeff Jirsa


 On Sep 2, 2017, at 7:10 AM, qf zhou  wrote:


 Yes, your are right. I am using STCS compaction strategy with some kind
 of timeseries model. Too much disk space has been occupied.

 What should I  do to stop  the  disk full ?

  I only want to keep 100 days data most recently,  so I set 
 default_time_to_live
 = 864(100 days ).

 I know I need to do something to stop the disk space cost, but I really
 don’t know how to do it.


 Here is the strategy of the big data table :

 AND compaction = {'class': 'org.apache.cassandra.db.compa
 ction.SizeTieredCompactionStrategy', 'max_threshold': '32',
 'min_threshold': '12', 'tombstone_threshold': '0.1',
 'unchecked_tombstone_compaction': 'true'}
 AND compression = {'chunk_length_in_kb': '64', 'class': '
 org.apache.cassandra.io.compress.LZ4Compressor'}
 AND crc_check_chance = 1.0
 AND dclocal_read_repair_chance = 0.1
 AND default_time_to_live = 864
 AND gc_grace_seconds = 432000



 在 2017年9月2日，下午7:34，Nicolas Guyomar  写道：

 your are using STCS compaction strategy with some kind of timeseries
 model, and you are going to end up with yor disk full!



>>>
>>
>> This message may contain confidential and/or privileged information.
>> If you are not the addressee or authorized to receive this on behalf of
>> the addressee you must not use, copy, disclose or take action based on this
>> message or any information herein.
>> If you have received this message in error, please advise the sender
>> immediately by reply email and delete this message. Thank you.
>>
>
>

Re: old big tombstone data file occupy much disk space

[Nicolas Guyomar]

You'll get the WARN "Will not compact {}: it is not an active sstable"  :)

On 4 September 2017 at 12:07, Shalom Sagges  wrote:

> By the way, does anyone know what happens if I run a user defined
> compaction on an sstable that's already in compaction?
>
>
>
>
>
>
> On Sun, Sep 3, 2017 at 2:55 PM, Shalom Sagges 
> wrote:
>
>> Try this blog by The Last Pickle:
>>
>> http://thelastpickle.com/blog/2016/10/18/user-defined-compaction.html
>>
>>
>>
>>
>>
>>
>> Shalom Sagges
>> DBA
>>  
>>  We Create Meaningful Connections
>>
>>
>>
>> On Sat, Sep 2, 2017 at 8:34 PM, Jeff Jirsa  wrote:
>>
>>> If you're on 3.0 (3.0.6 or 3.0.8 or newer I don't remember which), TWCS
>>> was designed for ttl-only time series use cases
>>>
>>> Alternatively, if you have IO to spare, you may find LCS works as well
>>> (it'll cause quite a bit more compaction, but a much higher chance to
>>> compact away tombstones)
>>>
>>> There are also tombstone focused sub properties to more aggressively
>>> compact sstables that have a lot of tombstones - check the docs for
>>> "unchecked tombstone compaction" and "tombstone threshold" - enabling those
>>> will enable more aggressive automatic single-sstable compactions
>>>
>>> --
>>> Jeff Jirsa
>>>
>>>
>>> On Sep 2, 2017, at 7:10 AM, qf zhou  wrote:
>>>
>>>
>>> Yes, your are right. I am using STCS compaction strategy with some kind
>>> of timeseries model. Too much disk space has been occupied.
>>>
>>> What should I  do to stop  the  disk full ?
>>>
>>>  I only want to keep 100 days data most recently,  so I set 
>>> default_time_to_live
>>> = 864(100 days ).
>>>
>>> I know I need to do something to stop the disk space cost, but I really
>>> don’t know how to do it.
>>>
>>>
>>> Here is the strategy of the big data table :
>>>
>>> AND compaction = {'class': 'org.apache.cassandra.db.compa
>>> ction.SizeTieredCompactionStrategy', 'max_threshold': '32',
>>> 'min_threshold': '12', 'tombstone_threshold': '0.1',
>>> 'unchecked_tombstone_compaction': 'true'}
>>> AND compression = {'chunk_length_in_kb': '64', 'class': '
>>> org.apache.cassandra.io.compress.LZ4Compressor'}
>>> AND crc_check_chance = 1.0
>>> AND dclocal_read_repair_chance = 0.1
>>> AND default_time_to_live = 864
>>> AND gc_grace_seconds = 432000
>>>
>>>
>>>
>>> 在 2017年9月2日，下午7:34，Nicolas Guyomar  写道：
>>>
>>> your are using STCS compaction strategy with some kind of timeseries
>>> model, and you are going to end up with yor disk full!
>>>
>>>
>>>
>>
>
> This message may contain confidential and/or privileged information.
> If you are not the addressee or authorized to receive this on behalf of
> the addressee you must not use, copy, disclose or take action based on this
> message or any information herein.
> If you have received this message in error, please advise the sender
> immediately by reply email and delete this message. Thank you.
>

Re: old big tombstone data file occupy much disk space

[Shalom Sagges]

By the way, does anyone know what happens if I run a user defined
compaction on an sstable that's already in compaction?





On Sun, Sep 3, 2017 at 2:55 PM, Shalom Sagges 
wrote:

> Try this blog by The Last Pickle:
>
> http://thelastpickle.com/blog/2016/10/18/user-defined-compaction.html
>
>
>
>
>
>
> Shalom Sagges
> DBA
>  
>  We Create Meaningful Connections
>
>
>
> On Sat, Sep 2, 2017 at 8:34 PM, Jeff Jirsa  wrote:
>
>> If you're on 3.0 (3.0.6 or 3.0.8 or newer I don't remember which), TWCS
>> was designed for ttl-only time series use cases
>>
>> Alternatively, if you have IO to spare, you may find LCS works as well
>> (it'll cause quite a bit more compaction, but a much higher chance to
>> compact away tombstones)
>>
>> There are also tombstone focused sub properties to more aggressively
>> compact sstables that have a lot of tombstones - check the docs for
>> "unchecked tombstone compaction" and "tombstone threshold" - enabling those
>> will enable more aggressive automatic single-sstable compactions
>>
>> --
>> Jeff Jirsa
>>
>>
>> On Sep 2, 2017, at 7:10 AM, qf zhou  wrote:
>>
>>
>> Yes, your are right. I am using STCS compaction strategy with some kind
>> of timeseries model. Too much disk space has been occupied.
>>
>> What should I  do to stop  the  disk full ?
>>
>>  I only want to keep 100 days data most recently,  so I set 
>> default_time_to_live
>> = 864(100 days ).
>>
>> I know I need to do something to stop the disk space cost, but I really
>> don’t know how to do it.
>>
>>
>> Here is the strategy of the big data table :
>>
>> AND compaction = {'class': 'org.apache.cassandra.db.compa
>> ction.SizeTieredCompactionStrategy', 'max_threshold': '32',
>> 'min_threshold': '12', 'tombstone_threshold': '0.1',
>> 'unchecked_tombstone_compaction': 'true'}
>> AND compression = {'chunk_length_in_kb': '64', 'class': '
>> org.apache.cassandra.io.compress.LZ4Compressor'}
>> AND crc_check_chance = 1.0
>> AND dclocal_read_repair_chance = 0.1
>> AND default_time_to_live = 864
>> AND gc_grace_seconds = 432000
>>
>>
>>
>> 在 2017年9月2日，下午7:34，Nicolas Guyomar  写道：
>>
>> your are using STCS compaction strategy with some kind of timeseries
>> model, and you are going to end up with yor disk full!
>>
>>
>>
>

-- 
This message may contain confidential and/or privileged information. 
If you are not the addressee or authorized to receive this on behalf of the 
addressee you must not use, copy, disclose or take action based on this 
message or any information herein. 
If you have received this message in error, please advise the sender 
immediately by reply email and delete this message. Thank you.