Re: Time Series schema performance

[Affan Syed]

Haris,

Like all things in Cassandra, you will need to create a down-sample
normalized table. ie either run a cron over the raw table, or if using some
streaming solution like Flink/Storm/Spark, to extract aggregate values and
put them into your downsample data.

HTH

- Affan

On Tue, May 29, 2018 at 10:24 PM, Haris Altaf  wrote:

> Hi All,
> I have a related question. How do you down-sample your timeseries data?
>
>
> regards,
> Haris
>
> On Tue, 29 May 2018 at 22:11 Jonathan Haddad  wrote:
>
>> I wrote a post on this topic a while ago, might be worth reading over:
>> http://thelastpickle.com/blog/2017/08/02/time-series-data-
>> modeling-massive-scale.html
>> On Tue, May 29, 2018 at 8:02 AM Jeff Jirsa  wrote:
>>
>> > There’s a third option which is doing bucketing by time instead of by
>> hash, which tends to perform quite well if you’re using TWCS as it makes
>> it
>> quite likely that a read can be served by a single sstable
>>
>> > --
>> > Jeff Jirsa
>>
>>
>> > On May 29, 2018, at 6:49 AM, sujeet jog  wrote:
>>
>> > Folks,
>> > I have two alternatives for the time series schema i have, and wanted to
>> weigh of on one of the schema .
>>
>> > The query is given id, & timestamp, read the metrics associated with the
>> id
>>
>> > The records are inserted every 5 mins, and the number of id's = 2
>> million,
>> > so at every 5mins  it will be 2 million records that will be written.
>>
>> > Bucket Range  : 0 - 5K.
>>
>> > Schema 1 )
>>
>> > create table (
>> > id timeuuid,
>> > bucketid Int,
>> > date date,
>> > timestamp timestamp,
>> > metricName1   BigInt,
>> > metricName2 BigInt.
>> > ...
>> > .
>> > metricName300 BigInt,
>>
>> > Primary Key (( day, bucketid ) ,  id, timestamp)
>> > )
>>
>> > BucketId is just a murmur3 hash of the id  which acts as a splitter to
>> group id's in a partition
>>
>>
>> > Pros : -
>>
>> > Efficient write performance, since data is written to minimal partitions
>>
>> > Cons : -
>>
>> > While the first schema works best when queried programmatically, but is
>> a
>> bit inflexible If it has to be integrated with 3rd party BI tools like
>> tableau, bucket-id cannot be generated from tableau as it's not part of
>> the
>> view etc..
>>
>>
>> > Schema 2 )
>> > Same as above, without bucketid &  date.
>>
>> > Primary Key (id, timestamp )
>>
>> > Pros : -
>>
>> > BI tools don't need to generate bucket id lookups,
>>
>> > Cons :-
>> > Too many partitions are written every 5 mins,  say 2 million records
>> written in distinct 2 million partitions.
>>
>>
>>
>> > I believe writing this data to commit log is same in case of Schema 1 &
>> Schema 2 ) , but the actual performance bottleneck could be compaction,
>> since the data from memtable is transformed to ssTables often based on the
>> memory settings, and
>> > the header for every SSTable would maintain partitionIndex with
>>   byteoffsets,
>>
>> >   wanted to guage how bad can the performance of Schema-2 go with
>> respect
>> to Write/Compaction having to do many diskseeks.
>>
>> > compacting many tables but with too many partitionIndex entries because
>> of the high number of parititions ,  can this be a bottleneck ?..
>>
>> > Any indept performance explanation of Schema-2 would be very much
>> helpful
>>
>>
>> > Thanks,
>>
>>
>>
>>
>> --
>> Jon Haddad
>> http://www.rustyrazorblade.com
>> twitter: rustyrazorblade
>>
>> -
>> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
>> For additional commands, e-mail: user-h...@cassandra.apache.org
>>
>> --
> regards,
> Haris
>

Re: nodetool (2.1.18) - Xmx, ParallelGCThreads, High CPU usage

[Chris Lohfink]

Might be better to disable explicit gcs so the full gcs don’t even occur. It’s 
likely from the rmi dgc or directbytebuffers not any actual need to do gcs or 
the concurrent gc threads would be an issue as well.

Nodetool also has no excuse to use that big of a heap so it should have max 
size capped too (along with parallel and concurrent gc threads).

Chris

Sent from my iPhone

> On May 29, 2018, at 4:42 PM, kurt greaves  wrote:
> 
> Good to know. So that confirms it's just the GC threads causing problems.
> 
>> On Tue., 29 May 2018, 22:02 Steinmaurer, Thomas, 
>>  wrote:
>> Kurt,
>> 
>>  
>> 
>> in our test it also didn’t made a difference with the default number of GC 
>> Threads (43 on our large machine) and running with Xmx128M or XmX31G 
>> (derived from $MAX_HEAP_SIZE). For both Xmx, we saw the high CPU caused by 
>> nodetool.
>> 
>>  
>> 
>> Regards,
>> 
>> Thomas
>> 
>>  
>> 
>> From: kurt greaves [mailto:k...@instaclustr.com] 
>> Sent: Dienstag, 29. Mai 2018 13:06
>> To: User 
>> Subject: Re: nodetool (2.1.18) - Xmx, ParallelGCThreads, High CPU usage
>> 
>>  
>> 
>> Thanks Thomas. After a bit more research today I found that the whole 
>> $MAX_HEAP_SIZE issue isn't really a problem because we don't explicitly set 
>> -Xms so the minimum heapsize by default will be 256mb, which isn't hugely 
>> problematic, and it's unlikely more than that would get allocated.
>> 
>>  
>> 
>> On 29 May 2018 at 09:29, Steinmaurer, Thomas 
>>  wrote:
>> 
>> Hi Kurt,
>> 
>>  
>> 
>> thanks for pointing me to the Xmx issue.
>> 
>>  
>> 
>> JIRA + patch (for Linux only based on C* 3.11) for the parallel GC thread 
>> issue is available here: 
>> https://issues.apache.org/jira/browse/CASSANDRA-14475
>> 
>>  
>> 
>> Thanks,
>> 
>> Thomas
>> 
>>  
>> 
>> From: kurt greaves [mailto:k...@instaclustr.com] 
>> Sent: Dienstag, 29. Mai 2018 05:54
>> To: User 
>> Subject: Re: nodetool (2.1.18) - Xmx, ParallelGCThreads, High CPU usage
>> 
>>  
>> 
>> 1) nodetool is reusing the $MAX_HEAP_SIZE environment variable, thus if we 
>> are running Cassandra with e.g. Xmx31G, nodetool is started with Xmx31G as 
>> well
>> 
>> This was fixed in 3.0.11/3.10 in CASSANDRA-12739. Not sure why it didn't 
>> make it into 2.1/2.2.
>> 
>> 2) As -XX:ParallelGCThreads is not explicitly set upon startup, this 
>> basically defaults to a value dependent on the number of cores. In our case, 
>> with the machine above, the number of parallel GC threads for the JVM is set 
>> to 43!
>> 3) Test-wise, we have adapted the nodetool startup script in a way to get a 
>> Java Flight Recording file on JVM exit, thus with each nodetool invocation 
>> we can inspect a JFR file. Here we may have seen System.gc() calls (without 
>> visible knowledge where they come from), GC times for the entire JVM 
>> life-time (e.g. ~1min) showing high cpu. This happened for both Xmx128M 
>> (default as it seems) and Xmx31G
>>  
>> After explicitly setting -XX:ParallelGCThreads=1 in the nodetool startup 
>> script, CPU usage spikes by nodetool are entirely gone.
>>  
>> Is this something which has been already adapted/tackled in Cassandra 
>> versions > 2.1 or worth to be considered as some sort of RFC?
>> 
>> Can you create a JIRA for this (and a patch, if you like)? We should be 
>> explicitly setting this on nodetool invocations.
>> 
>> ​
>> 
>> The contents of this e-mail are intended for the named addressee only. It 
>> contains information that may be confidential. Unless you are the named 
>> addressee or an authorized designee, you may not copy or use it, or disclose 
>> it to anyone else. If you received it in error please notify us immediately 
>> and then destroy it. Dynatrace Austria GmbH (registration number FN 91482h) 
>> is a company registered in Linz whose registered office is at 4040 Linz, 
>> Austria, Freistädterstraße 313
>> 
>>  
>> 
>> The contents of this e-mail are intended for the named addressee only. It 
>> contains information that may be confidential. Unless you are the named 
>> addressee or an authorized designee, you may not copy or use it, or disclose 
>> it to anyone else. If you received it in error please notify us immediately 
>> and then destroy it. Dynatrace Austria GmbH (registration number FN 91482h) 
>> is a company registered in Linz whose registered office is at 4040 Linz, 
>> Austria, Freistädterstraße 313

Re: nodetool (2.1.18) - Xmx, ParallelGCThreads, High CPU usage

[kurt greaves]

Good to know. So that confirms it's just the GC threads causing problems.

On Tue., 29 May 2018, 22:02 Steinmaurer, Thomas, <
thomas.steinmau...@dynatrace.com> wrote:

> Kurt,
>
>
>
> in our test it also didn’t made a difference with the default number of GC
> Threads (43 on our large machine) and running with Xmx128M or XmX31G
> (derived from $MAX_HEAP_SIZE). For both Xmx, we saw the high CPU caused by
> nodetool.
>
>
>
> Regards,
>
> Thomas
>
>
>
> *From:* kurt greaves [mailto:k...@instaclustr.com]
> *Sent:* Dienstag, 29. Mai 2018 13:06
> *To:* User 
> *Subject:* Re: nodetool (2.1.18) - Xmx, ParallelGCThreads, High CPU usage
>
>
>
> Thanks Thomas. After a bit more research today I found that the whole
> $MAX_HEAP_SIZE issue isn't really a problem because we don't explicitly set
> -Xms so the minimum heapsize by default will be 256mb, which isn't hugely
> problematic, and it's unlikely more than that would get allocated.
>
>
>
> On 29 May 2018 at 09:29, Steinmaurer, Thomas <
> thomas.steinmau...@dynatrace.com> wrote:
>
> Hi Kurt,
>
>
>
> thanks for pointing me to the Xmx issue.
>
>
>
> JIRA + patch (for Linux only based on C* 3.11) for the parallel GC thread
> issue is available here:
> https://issues.apache.org/jira/browse/CASSANDRA-14475
>
>
>
> Thanks,
>
> Thomas
>
>
>
> *From:* kurt greaves [mailto:k...@instaclustr.com]
> *Sent:* Dienstag, 29. Mai 2018 05:54
> *To:* User 
> *Subject:* Re: nodetool (2.1.18) - Xmx, ParallelGCThreads, High CPU usage
>
>
>
> 1) nodetool is reusing the $MAX_HEAP_SIZE environment variable, thus if we
> are running Cassandra with e.g. Xmx31G, nodetool is started with Xmx31G as
> well
>
> This was fixed in 3.0.11/3.10 in CASSANDRA-12739
> . Not sure why it
> didn't make it into 2.1/2.2.
>
> 2) As -XX:ParallelGCThreads is not explicitly set upon startup, this
> basically defaults to a value dependent on the number of cores. In our
> case, with the machine above, the number of parallel GC threads for the JVM
> is set to 43!
> 3) Test-wise, we have adapted the nodetool startup script in a way to get
> a Java Flight Recording file on JVM exit, thus with each nodetool
> invocation we can inspect a JFR file. Here we may have seen System.gc()
> calls (without visible knowledge where they come from), GC times for the
> entire JVM life-time (e.g. ~1min) showing high cpu. This happened for both
> Xmx128M (default as it seems) and Xmx31G
>
> After explicitly setting -XX:ParallelGCThreads=1 in the nodetool startup
> script, CPU usage spikes by nodetool are entirely gone.
>
> Is this something which has been already adapted/tackled in Cassandra
> versions > 2.1 or worth to be considered as some sort of RFC?
>
> Can you create a JIRA for this (and a patch, if you like)? We should be
> explicitly setting this on nodetool invocations.
>
> ​
>
> The contents of this e-mail are intended for the named addressee only. It
> contains information that may be confidential. Unless you are the named
> addressee or an authorized designee, you may not copy or use it, or
> disclose it to anyone else. If you received it in error please notify us
> immediately and then destroy it. Dynatrace Austria GmbH (registration
> number FN 91482h) is a company registered in Linz whose registered office
> is at 4040 Linz, Austria, Freistädterstraße 313
>
>
> The contents of this e-mail are intended for the named addressee only. It
> contains information that may be confidential. Unless you are the named
> addressee or an authorized designee, you may not copy or use it, or
> disclose it to anyone else. If you received it in error please notify us
> immediately and then destroy it. Dynatrace Austria GmbH (registration
> number FN 91482h) is a company registered in Linz whose registered office
> is at 4040 Linz, Austria, Freistädterstraße 313
>

Re: Time Series schema performance

[Haris Altaf]

Hi All,
I have a related question. How do you down-sample your timeseries data?


regards,
Haris

On Tue, 29 May 2018 at 22:11 Jonathan Haddad  wrote:

> I wrote a post on this topic a while ago, might be worth reading over:
>
> http://thelastpickle.com/blog/2017/08/02/time-series-data-modeling-massive-scale.html
> On Tue, May 29, 2018 at 8:02 AM Jeff Jirsa  wrote:
>
> > There’s a third option which is doing bucketing by time instead of by
> hash, which tends to perform quite well if you’re using TWCS as it makes it
> quite likely that a read can be served by a single sstable
>
> > --
> > Jeff Jirsa
>
>
> > On May 29, 2018, at 6:49 AM, sujeet jog  wrote:
>
> > Folks,
> > I have two alternatives for the time series schema i have, and wanted to
> weigh of on one of the schema .
>
> > The query is given id, & timestamp, read the metrics associated with the
> id
>
> > The records are inserted every 5 mins, and the number of id's = 2
> million,
> > so at every 5mins  it will be 2 million records that will be written.
>
> > Bucket Range  : 0 - 5K.
>
> > Schema 1 )
>
> > create table (
> > id timeuuid,
> > bucketid Int,
> > date date,
> > timestamp timestamp,
> > metricName1   BigInt,
> > metricName2 BigInt.
> > ...
> > .
> > metricName300 BigInt,
>
> > Primary Key (( day, bucketid ) ,  id, timestamp)
> > )
>
> > BucketId is just a murmur3 hash of the id  which acts as a splitter to
> group id's in a partition
>
>
> > Pros : -
>
> > Efficient write performance, since data is written to minimal partitions
>
> > Cons : -
>
> > While the first schema works best when queried programmatically, but is a
> bit inflexible If it has to be integrated with 3rd party BI tools like
> tableau, bucket-id cannot be generated from tableau as it's not part of the
> view etc..
>
>
> > Schema 2 )
> > Same as above, without bucketid &  date.
>
> > Primary Key (id, timestamp )
>
> > Pros : -
>
> > BI tools don't need to generate bucket id lookups,
>
> > Cons :-
> > Too many partitions are written every 5 mins,  say 2 million records
> written in distinct 2 million partitions.
>
>
>
> > I believe writing this data to commit log is same in case of Schema 1 &
> Schema 2 ) , but the actual performance bottleneck could be compaction,
> since the data from memtable is transformed to ssTables often based on the
> memory settings, and
> > the header for every SSTable would maintain partitionIndex with
>   byteoffsets,
>
> >   wanted to guage how bad can the performance of Schema-2 go with respect
> to Write/Compaction having to do many diskseeks.
>
> > compacting many tables but with too many partitionIndex entries because
> of the high number of parititions ,  can this be a bottleneck ?..
>
> > Any indept performance explanation of Schema-2 would be very much helpful
>
>
> > Thanks,
>
>
>
>
> --
> Jon Haddad
> http://www.rustyrazorblade.com
> twitter: rustyrazorblade
>
> -
> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
> For additional commands, e-mail: user-h...@cassandra.apache.org
>
> --
regards,
Haris

Re: Time Series schema performance

[Jonathan Haddad]

I wrote a post on this topic a while ago, might be worth reading over:
http://thelastpickle.com/blog/2017/08/02/time-series-data-modeling-massive-scale.html
On Tue, May 29, 2018 at 8:02 AM Jeff Jirsa  wrote:

> There’s a third option which is doing bucketing by time instead of by
hash, which tends to perform quite well if you’re using TWCS as it makes it
quite likely that a read can be served by a single sstable

> --
> Jeff Jirsa


> On May 29, 2018, at 6:49 AM, sujeet jog  wrote:

> Folks,
> I have two alternatives for the time series schema i have, and wanted to
weigh of on one of the schema .

> The query is given id, & timestamp, read the metrics associated with the
id

> The records are inserted every 5 mins, and the number of id's = 2
million,
> so at every 5mins  it will be 2 million records that will be written.

> Bucket Range  : 0 - 5K.

> Schema 1 )

> create table (
> id timeuuid,
> bucketid Int,
> date date,
> timestamp timestamp,
> metricName1   BigInt,
> metricName2 BigInt.
> ...
> .
> metricName300 BigInt,

> Primary Key (( day, bucketid ) ,  id, timestamp)
> )

> BucketId is just a murmur3 hash of the id  which acts as a splitter to
group id's in a partition


> Pros : -

> Efficient write performance, since data is written to minimal partitions

> Cons : -

> While the first schema works best when queried programmatically, but is a
bit inflexible If it has to be integrated with 3rd party BI tools like
tableau, bucket-id cannot be generated from tableau as it's not part of the
view etc..


> Schema 2 )
> Same as above, without bucketid &  date.

> Primary Key (id, timestamp )

> Pros : -

> BI tools don't need to generate bucket id lookups,

> Cons :-
> Too many partitions are written every 5 mins,  say 2 million records
written in distinct 2 million partitions.



> I believe writing this data to commit log is same in case of Schema 1 &
Schema 2 ) , but the actual performance bottleneck could be compaction,
since the data from memtable is transformed to ssTables often based on the
memory settings, and
> the header for every SSTable would maintain partitionIndex with
  byteoffsets,

>   wanted to guage how bad can the performance of Schema-2 go with respect
to Write/Compaction having to do many diskseeks.

> compacting many tables but with too many partitionIndex entries because
of the high number of parititions ,  can this be a bottleneck ?..

> Any indept performance explanation of Schema-2 would be very much helpful


> Thanks,




-- 
Jon Haddad
http://www.rustyrazorblade.com
twitter: rustyrazorblade

-
To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
For additional commands, e-mail: user-h...@cassandra.apache.org

Re: Time Series schema performance

[Jeff Jirsa]

There’s a third option which is doing bucketing by time instead of by hash, 
which tends to perform quite well if you’re using TWCS as it makes it quite 
likely that a read can be served by a single sstable

-- 
Jeff Jirsa


> On May 29, 2018, at 6:49 AM, sujeet jog  wrote:
> 
> Folks, 
> I have two alternatives for the time series schema i have, and wanted to 
> weigh of on one of the schema . 
> 
> The query is given id, & timestamp, read the metrics associated with the id
> 
> The records are inserted every 5 mins, and the number of id's = 2 million, 
> so at every 5mins  it will be 2 million records that will be written.  
> 
> Bucket Range  : 0 - 5K.
> 
> Schema 1 ) 
> 
> create table (
> id timeuuid,
> bucketid Int, 
> date date,
> timestamp timestamp, 
> metricName1   BigInt,
> metricName2 BigInt. 
> ...
> .
> metricName300 BigInt,
> 
> Primary Key (( day, bucketid ) ,  id, timestamp)
> ) 
> 
> BucketId is just a murmur3 hash of the id  which acts as a splitter to group 
> id's in a partition
> 
> 
> Pros : - 
> 
> Efficient write performance, since data is written to minimal partitions
> 
> Cons : -
> 
> While the first schema works best when queried programmatically, but is a bit 
> inflexible If it has to be integrated with 3rd party BI tools like tableau, 
> bucket-id cannot be generated from tableau as it's not part of the view etc..
> 
> 
> Schema 2 ) 
> Same as above, without bucketid &  date. 
> 
> Primary Key (id, timestamp ) 
> 
> Pros : -
> 
> BI tools don't need to generate bucket id lookups, 
> 
> Cons :-
> Too many partitions are written every 5 mins,  say 2 million records written 
> in distinct 2 million partitions.
> 
> 
> 
> I believe writing this data to commit log is same in case of Schema 1 & 
> Schema 2 ) , but the actual performance bottleneck could be compaction, since 
> the data from memtable is transformed to ssTables often based on the memory 
> settings, and 
> the header for every SSTable would maintain partitionIndex with  byteoffsets, 
> 
>  wanted to guage how bad can the performance of Schema-2 go with respect to 
> Write/Compaction having to do many diskseeks.
> 
> compacting many tables but with too many partitionIndex entries because of 
> the high number of parititions ,  can this be a bottleneck ?..
> 
> Any indept performance explanation of Schema-2 would be very much helpful
> 
> 
> Thanks, 
> 
> 

Time Series schema performance

[sujeet jog]

Folks,
I have two alternatives for the time series schema i have, and wanted to
weigh of on one of the schema .

The query is given id, & timestamp, read the metrics associated with the id

The records are inserted every 5 mins, and the number of id's = 2 million,
so at every 5mins  it will be 2 million records that will be written.

Bucket Range  : 0 - 5K.

Schema 1 )

create table (
id timeuuid,
bucketid Int,
date date,
timestamp timestamp,
metricName1   BigInt,
metricName2 BigInt.
...
.
metricName300 BigInt,

Primary Key (( day, bucketid ) ,  id, timestamp)
)

BucketId is just a murmur3 hash of the id  which acts as a splitter to
group id's in a partition


Pros : -

Efficient write performance, since data is written to minimal partitions

Cons : -

While the first schema works best when queried programmatically, but is a
bit inflexible If it has to be integrated with 3rd party BI tools like
tableau, bucket-id cannot be generated from tableau as it's not part of the
view etc..


Schema 2 )
Same as above, without bucketid &  date.

Primary Key (id, timestamp )

Pros : -

BI tools don't need to generate bucket id lookups,

Cons :-
Too many partitions are written every 5 mins,  say 2 million records
written in distinct 2 million partitions.



I believe writing this data to commit log is same in case of Schema 1 &
Schema 2 ) , but the actual performance bottleneck could be compaction,
since the data from memtable is transformed to ssTables often based on the
memory settings, and
the header for every SSTable would maintain partitionIndex with
byteoffsets,

 wanted to guage how bad can the performance of Schema-2 go with respect to
Write/Compaction having to do many diskseeks.

compacting many tables but with too many partitionIndex entries because of
the high number of parititions ,  can this be a bottleneck ?..

Any indept performance explanation of Schema-2 would be very much helpful


Thanks,

RE: nodetool (2.1.18) - Xmx, ParallelGCThreads, High CPU usage

[Steinmaurer, Thomas]

Kurt,

in our test it also didn’t made a difference with the default number of GC 
Threads (43 on our large machine) and running with Xmx128M or XmX31G (derived 
from $MAX_HEAP_SIZE). For both Xmx, we saw the high CPU caused by nodetool.

Regards,
Thomas

From: kurt greaves [mailto:k...@instaclustr.com]
Sent: Dienstag, 29. Mai 2018 13:06
To: User 
Subject: Re: nodetool (2.1.18) - Xmx, ParallelGCThreads, High CPU usage

Thanks Thomas. After a bit more research today I found that the whole 
$MAX_HEAP_SIZE issue isn't really a problem because we don't explicitly set 
-Xms so the minimum heapsize by default will be 256mb, which isn't hugely 
problematic, and it's unlikely more than that would get allocated.

On 29 May 2018 at 09:29, Steinmaurer, Thomas 
mailto:thomas.steinmau...@dynatrace.com>> 
wrote:
Hi Kurt,

thanks for pointing me to the Xmx issue.

JIRA + patch (for Linux only based on C* 3.11) for the parallel GC thread issue 
is available here: https://issues.apache.org/jira/browse/CASSANDRA-14475

Thanks,
Thomas

From: kurt greaves [mailto:k...@instaclustr.com]
Sent: Dienstag, 29. Mai 2018 05:54
To: User mailto:user@cassandra.apache.org>>
Subject: Re: nodetool (2.1.18) - Xmx, ParallelGCThreads, High CPU usage

1) nodetool is reusing the $MAX_HEAP_SIZE environment variable, thus if we are 
running Cassandra with e.g. Xmx31G, nodetool is started with Xmx31G as well
This was fixed in 3.0.11/3.10 in 
CASSANDRA-12739. Not 
sure why it didn't make it into 2.1/2.2.
2) As -XX:ParallelGCThreads is not explicitly set upon startup, this basically 
defaults to a value dependent on the number of cores. In our case, with the 
machine above, the number of parallel GC threads for the JVM is set to 43!
3) Test-wise, we have adapted the nodetool startup script in a way to get a 
Java Flight Recording file on JVM exit, thus with each nodetool invocation we 
can inspect a JFR file. Here we may have seen System.gc() calls (without 
visible knowledge where they come from), GC times for the entire JVM life-time 
(e.g. ~1min) showing high cpu. This happened for both Xmx128M (default as it 
seems) and Xmx31G

After explicitly setting -XX:ParallelGCThreads=1 in the nodetool startup 
script, CPU usage spikes by nodetool are entirely gone.

Is this something which has been already adapted/tackled in Cassandra versions 
> 2.1 or worth to be considered as some sort of RFC?
Can you create a JIRA for this (and a patch, if you like)? We should be 
explicitly setting this on nodetool invocations.
​
The contents of this e-mail are intended for the named addressee only. It 
contains information that may be confidential. Unless you are the named 
addressee or an authorized designee, you may not copy or use it, or disclose it 
to anyone else. If you received it in error please notify us immediately and 
then destroy it. Dynatrace Austria GmbH (registration number FN 91482h) is a 
company registered in Linz whose registered office is at 4040 Linz, Austria, 
Freistädterstraße 313

The contents of this e-mail are intended for the named addressee only. It 
contains information that may be confidential. Unless you are the named 
addressee or an authorized designee, you may not copy or use it, or disclose it 
to anyone else. If you received it in error please notify us immediately and 
then destroy it. Dynatrace Austria GmbH (registration number FN 91482h) is a 
company registered in Linz whose registered office is at 4040 Linz, Austria, 
Freistädterstraße 313

Re: nodetool (2.1.18) - Xmx, ParallelGCThreads, High CPU usage

[kurt greaves]

Thanks Thomas. After a bit more research today I found that the whole
$MAX_HEAP_SIZE issue isn't really a problem because we don't explicitly set
-Xms so the minimum heapsize by default will be 256mb, which isn't hugely
problematic, and it's unlikely more than that would get allocated.

On 29 May 2018 at 09:29, Steinmaurer, Thomas <
thomas.steinmau...@dynatrace.com> wrote:

> Hi Kurt,
>
>
>
> thanks for pointing me to the Xmx issue.
>
>
>
> JIRA + patch (for Linux only based on C* 3.11) for the parallel GC thread
> issue is available here: https://issues.apache.org/
> jira/browse/CASSANDRA-14475
>
>
>
> Thanks,
>
> Thomas
>
>
>
> *From:* kurt greaves [mailto:k...@instaclustr.com]
> *Sent:* Dienstag, 29. Mai 2018 05:54
> *To:* User 
> *Subject:* Re: nodetool (2.1.18) - Xmx, ParallelGCThreads, High CPU usage
>
>
>
> 1) nodetool is reusing the $MAX_HEAP_SIZE environment variable, thus if we
> are running Cassandra with e.g. Xmx31G, nodetool is started with Xmx31G as
> well
>
> This was fixed in 3.0.11/3.10 in CASSANDRA-12739
> . Not sure why it
> didn't make it into 2.1/2.2.
>
> 2) As -XX:ParallelGCThreads is not explicitly set upon startup, this
> basically defaults to a value dependent on the number of cores. In our
> case, with the machine above, the number of parallel GC threads for the JVM
> is set to 43!
> 3) Test-wise, we have adapted the nodetool startup script in a way to get
> a Java Flight Recording file on JVM exit, thus with each nodetool
> invocation we can inspect a JFR file. Here we may have seen System.gc()
> calls (without visible knowledge where they come from), GC times for the
> entire JVM life-time (e.g. ~1min) showing high cpu. This happened for both
> Xmx128M (default as it seems) and Xmx31G
>
> After explicitly setting -XX:ParallelGCThreads=1 in the nodetool startup
> script, CPU usage spikes by nodetool are entirely gone.
>
> Is this something which has been already adapted/tackled in Cassandra
> versions > 2.1 or worth to be considered as some sort of RFC?
>
> Can you create a JIRA for this (and a patch, if you like)? We should be
> explicitly setting this on nodetool invocations.
>
> ​
> The contents of this e-mail are intended for the named addressee only. It
> contains information that may be confidential. Unless you are the named
> addressee or an authorized designee, you may not copy or use it, or
> disclose it to anyone else. If you received it in error please notify us
> immediately and then destroy it. Dynatrace Austria GmbH (registration
> number FN 91482h) is a company registered in Linz whose registered office
> is at 4040 Linz, Austria, Freistädterstraße 313
>

Scaling in Cassandra

[Vishal1.Sharma]

Dear Community,

I’ll appreciate if I can get some help below:

https://stackoverflow.com/q/50581473/5701173

Regards,
Vishal Sharma
"Confidentiality Warning: This message and any attachments are intended only 
for the use of the intended recipient(s). 
are confidential and may be privileged. If you are not the intended recipient. 
you are hereby notified that any 
review. re-transmission. conversion to hard copy. copying. circulation or other 
use of this message and any attachments is 
strictly prohibited. If you are not the intended recipient. please notify the 
sender immediately by return email. 
and delete this message and any attachments from your system.

Virus Warning: Although the company has taken reasonable precautions to ensure 
no viruses are present in this email. 
The company cannot accept responsibility for any loss or damage arising from 
the use of this email or attachment."

RE: nodetool (2.1.18) - Xmx, ParallelGCThreads, High CPU usage

[Steinmaurer, Thomas]

Hi Kurt,

thanks for pointing me to the Xmx issue.

JIRA + patch (for Linux only based on C* 3.11) for the parallel GC thread issue 
is available here: https://issues.apache.org/jira/browse/CASSANDRA-14475

Thanks,
Thomas

From: kurt greaves [mailto:k...@instaclustr.com]
Sent: Dienstag, 29. Mai 2018 05:54
To: User 
Subject: Re: nodetool (2.1.18) - Xmx, ParallelGCThreads, High CPU usage

1) nodetool is reusing the $MAX_HEAP_SIZE environment variable, thus if we are 
running Cassandra with e.g. Xmx31G, nodetool is started with Xmx31G as well
This was fixed in 3.0.11/3.10 in 
CASSANDRA-12739. Not 
sure why it didn't make it into 2.1/2.2.
2) As -XX:ParallelGCThreads is not explicitly set upon startup, this basically 
defaults to a value dependent on the number of cores. In our case, with the 
machine above, the number of parallel GC threads for the JVM is set to 43!
3) Test-wise, we have adapted the nodetool startup script in a way to get a 
Java Flight Recording file on JVM exit, thus with each nodetool invocation we 
can inspect a JFR file. Here we may have seen System.gc() calls (without 
visible knowledge where they come from), GC times for the entire JVM life-time 
(e.g. ~1min) showing high cpu. This happened for both Xmx128M (default as it 
seems) and Xmx31G

After explicitly setting -XX:ParallelGCThreads=1 in the nodetool startup 
script, CPU usage spikes by nodetool are entirely gone.

Is this something which has been already adapted/tackled in Cassandra versions 
> 2.1 or worth to be considered as some sort of RFC?
Can you create a JIRA for this (and a patch, if you like)? We should be 
explicitly setting this on nodetool invocations.
​
The contents of this e-mail are intended for the named addressee only. It 
contains information that may be confidential. Unless you are the named 
addressee or an authorized designee, you may not copy or use it, or disclose it 
to anyone else. If you received it in error please notify us immediately and 
then destroy it. Dynatrace Austria GmbH (registration number FN 91482h) is a 
company registered in Linz whose registered office is at 4040 Linz, Austria, 
Freistädterstraße 313

Re: Certified Cassandra for Enterprise use

[Ben Slater]

Hi Pranay

We (Instaclustr) provide enterprise support for Cassandra (
https://www.instaclustr.com/services/cassandra-support/) which may cover
what you are looking for.

Please get in touch direct if you would like to discuss.

Cheers
Ben

On Tue, 29 May 2018 at 10:11 Pranay akula 
wrote:

> Is there any third party who provides security patches/releases for Apache
> cassandra
>
> For Enterprise use is there any third party who provides certified Apache
> cassandra packages ??
>
> Thanks
> Pranay
>
-- 


*Ben Slater*

*Chief Product Officer *

   


Read our latest technical blog posts here
.

This email has been sent on behalf of Instaclustr Pty. Limited (Australia)
and Instaclustr Inc (USA).

This email and any attachments may contain confidential and legally
privileged information.  If you are not the intended recipient, do not copy
or disclose its content, but please reply to this email immediately and
highlight the error to the sender and then immediately delete the message.

Re: Using K8s to Manage Cassandra in Production

[Hassaan Pasha]

Thank you everyone.
This thread has been really useful!

On Wed, May 23, 2018 at 8:59 PM, Ben Bromhead  wrote:

> Here is the expectations around compatibility levels https://github.com/
> kubernetes/community/blob/master/contributors/design-
> proposals/api-machinery/csi-new-client-library-procedure.
> md#client-capabilities Though references to gold, silver, bronze etc seem
> to have largely gone away... not sure what's going on there?
>
> For a full reference just browser through the repo, https://github.com/
> kubernetes-client/java/blob/master/kubernetes/README.md is a good place
> to start as is https://github.com/kubernetes-client/java/tree/
> master/examples
>
> The Java driver doesn't have as much of the nice things in
> https://github.com/kubernetes/client-go/tree/master/tools but it does
> have some good helper classes in the util package so I guess we spent a
> little more time wiring things together?
>
> Code Generation is done via the jsonschema2pojo maven plugin and we also
> just have the raw CRD definition in a resource directory.
>
> On Wed, May 23, 2018 at 11:23 AM vincent gromakowski <
> vincent.gromakow...@gmail.com> wrote:
>
>> Thanks ! Do you have some pointers on the available features ? I am more
>> afraid of the lack of custom controller integration, for instance the code
>> generator...
>>
>> 2018-05-23 17:17 GMT+02:00 Ben Bromhead :
>>
>>> The official Kubernetes Java driver is actually pretty feature complete,
>>> if not exactly idiomatic Java...  it's only missing full examples to get it
>>> to GOLD compatibility levels iirc.
>>>
>>> A few reasons we went down the Java path:
>>>
>>>- Cassandra community engagement was the primary concern. If you are
>>>a developer in the Cassandra community you have a base level of Java
>>>knowledge, so it means if you want to work on the Kubernetes operator you
>>>only have to learn 1 thing, Kubernetes. If the operator was in Go,
>>>you would then have two things to learn, Go and Kubernetes :)
>>>- We actually wrote an initial PoC in Go (based off the etcd
>>>operator, you can find it here https://github.com/
>>>benbromhead/cassandra-operator-old
>>> ), but
>>>because it was in Go we ended up making architectural decisions simply
>>>because Go doesn't do JMX, so it felt like we were just fighting 
>>> different
>>>ecosystems just to be part of the cool group.
>>>
>>> Some other less important points weighed the decision in Java's favour:
>>>
>>>- The folk at Instaclustr all know Java, and are productive in it
>>>from day 1. Go is fun and relatively simple, but not our forte.
>>>-  Mature package management, Generics/inability to write DRY
>>>code, a million if err statements  (:
>>>- Some other awesome operators/controllers are written in JVM based
>>>languages. The sparkKubernetes resource manager (which is a k8s 
>>> controller)
>>>is written in Scala.
>>>
>>>
>>> On Wed, May 23, 2018 at 10:04 AM vincent gromakowski <
>>> vincent.gromakow...@gmail.com> wrote:
>>>
 Why did you choose java for the operator implementation when everybody
 seems to use the go client (probably for greater functionalities) ?

 2018-05-23 15:39 GMT+02:00 Ben Bromhead :

> You can get a good way with StatefulSets, but as Tom mentioned there
> are still some issues with this, particularly around scaling up and down.
>
> We are working on an Operator for Apache Cassandra, you can find it
> here https://github.com/instaclustr/cassandra-operator. This is a
> joint project between Instaclustr, Pivotal and a few other folk.
>
> Currently it's a work in progress, but we would love any or all early
> feedback/PRs/issues etc. Our first GA release will target the following
> capabilities:
>
>- Safe scaling up and down (including decommissioning)
>- Backup/restore workflow (snapshots only initially)
>- Built in prometheus integration and discovery
>
> Other features like repair, better PV support, maybe even a nice
> dashboard will be on the way.
>
>
> On Wed, May 23, 2018 at 7:35 AM Tom Petracca 
> wrote:
>
>> Using a statefulset should get you pretty far, though will likely be
>> less effective than a coreos-style “operator”. Some random points:
>>
>>- For scale-up: a node shouldn’t report “ready” until it’s in the
>>NORMAL state; this will prevent multiple nodes from bootstrapping at 
>> once.
>>- For scale-down: as of now there isn’t a mechanism to know if a
>>pod is getting decommissioned because you’ve permanently lowered 
>> replica
>>count, or because it’s just getting bounced/re-scheduled, thus knowing
>>whether or not to decommission is basically impossible. Relevant 
>> issue:
>>kubernetes/kubernetes#1462
>>