Re: Copy from CSV on OS X problem with varint values <= -2^63

[Stefania Alborghetti]

It doesn't look like the embedded driver, it should come from a zip file
labeled with version *3.7.0.post0-2481531* for cassandra 3.10:

*Using CQL driver: *

Sorry, I should have posted this example in my previous email, rather than
an example based on the non-embedded driver.

I don't know who to contact regarding homebrew installation, but you could
download  the Cassandra package,
unzip it, and run cqlsh and Cassandra from that directory?


On Thu, Apr 6, 2017 at 4:59 AM, Boris Babic  wrote:

> Stefania
>
> This is the output of my --debug, I never touched CQLSH_NO_BUNDLED and did
> not know about it.
> As you can see I have used homebrew to install Cassandra and looks like
> its the embedded version as it sits under the Cassandra folder ?
>
> cqlsh --debug
> Using CQL driver:  cassandra/3.10_1/libexec/vendor/lib/python2.7/site-
> packages/cassandra/__init__.pyc'>
> Using connect timeout: 5 seconds
> Using 'utf-8' encoding
> Using ssl: False
> Connected to Test Cluster at 127.0.0.1:9042.
> [cqlsh 5.0.1 | Cassandra 3.10 | CQL spec 3.4.4 | Native protocol v4]
> Use HELP for help.
>
>
> On Apr 5, 2017, at 12:07 PM, Stefania Alborghetti  datastax.com> wrote:
>
> You are welcome.
>
> I traced the problem to a commit of the Python driver that shipped in
> version 3.8 of the driver. It is fixed in 3.8.1. More details
> on CASSANDRA-13408. I don't think it's related to the OS.
>
> Since Cassandra 3.10 ships with an older version of the driver embedded in
> a zip file in the lib folder, and this version is not affected, I'm
> guessing that either the embedded version does not work on OS X, or you are
> manually using a different version of the driver by
> setting CQLSH_NO_BUNDLED (which is why I could reproduce it on my laptop).
>
> You can run cqlsh with --debug to see the version of the driver that cqlsh
> is using, for example:
>
> cqlsh --debug
> Using CQL driver:  dist-packages/cassandra_driver-3.8.1-py2.7-linux-x86_
> 64.egg/cassandra/__init__.pyc'>
>
> Can you confirm if you were overriding the Python driver by
> setting CQLSH_NO_BUNDLED and the version of the driver?
>
>
>
> On Tue, Apr 4, 2017 at 6:12 PM, Boris Babic  wrote:
> Thanks Stefania, going from memory don't think I noticed this on windows
> but haven't got a machine handy to test it on at the moment.
>
> On Apr 4, 2017, at 19:44, Stefania Alborghetti  datastax.com> wrote:
>
> I've reproduced the same problem on Linux, and I've
> opened CASSANDRA-13408. As a workaround, disable prepared statements and it
> will work (WITH HEADER = TRUE AND PREPAREDSTATEMENTS = False).
>
> On Tue, Apr 4, 2017 at 5:02 PM, Boris Babic  wrote:
>
> On Apr 4, 2017, at 7:00 PM, Boris Babic  wrote:
>
> Hi
>
> I’m testing the write of various datatypes on OS X for fun running
> cassandra 3.10 on a single laptop instance, and from what i can see
> varint should map to java.math.BigInteger and have no problems with
> Long.MIN_VALE , -9223372036854775808, but i can’t see what I’m doing wrong.
>
> cqlsh: 5.0.1
> cassandra 3.10
> osx el capitan.
>
> data.csv:
>
> id,varint
> -2147483648 <(214)%20748-3648>,-9223372036854775808
> 2147483647 <(214)%20748-3647>,9223372036854775807
>
> COPY mykeyspace.data (id,varint) FROM 'data.csv' WITH HEADER=true;
>
>   Failed to make batch statement: Received an argument of invalid type
> for column "varint". Expected: ,
> Got: ; (descriptor 'bit_length' requires a 'int' object but
> received a 'long’)
>
> If I directly type a similar insert in cqlsh no such problem occurs, in
> fact I can make the value many orders of magnitude less and all is fine.
>
> cqlsh> insert into mykeyspace.data (id,varint) 
> values(1,-9223372036854775808898989898)
> ;
>
> Had not observed this before on other OS, is this something todo with the
> way the copy from parser is interpreting varint for values <= -2^63 ?
>
> Thanks for any input
> Boris
>
>
>
>
>
>
>
>
>
>
>
>
> --
>
> STEFANIA ALBORGHETTI
> Software engineer | +852 6114 9265 <+852%206114%209265> |
> stefania.alborghe...@datastax.com
>
> [image: http://www.datastax.com/cloud-applications]
>
>
>
>
>
> --
>
> STEFANIA ALBORGHETTI
> Software engineer | +852 6114 9265 <+852%206114%209265> |
> stefania.alborghe...@datastax.com
>
> [image: http://www.datastax.com/cloud-applications]
>
>
>
>


-- 



STEFANIA ALBORGHETTI

Software engineer | +852 6114 9265 | stefania.alborghe...@datastax.com


[image: http://www.datastax.com/cloud-applications]

Re: how to recover a dead node using commit log when memtable is lost

[preetika tyagi]

Assuming we are using periodic mode for commit log sync.

On Wed, Apr 5, 2017 at 3:29 PM, preetika tyagi 
wrote:

> Very good explanation.
> One follow-up question. If CL is set to 1 and RF to 3, then there are
> chances of the data being lost if the machine crashes before replication
> happens and the commit log (on the node which processed the data for CF=1)
> is not synced yet. Right?
>
> Thanks,
> Preetika
>
> On Wed, Apr 5, 2017 at 1:17 PM, Bhuvan Rawal  wrote:
>
>> I beg to differ with @Matija here, IMO by default cassandra syncs data
>> into commit log in a periodic fashion with a fsync period of 10 sec (Ref -
>> https://github.com/apache/cassandra/blob/trunk/conf/cassandra.yaml#L361).
>> If a write is not written to disk and RF is 1 else CL is Local One & node
>> goes down then there could be potential data loss and client would expect
>> data to be present.
>>
>> Therefore a good strategy would be to either have RF 3 and write with
>> quorum. Or if thats not a feasible option then use Batch Mode for commitlog
>> sync - That could lead to much higher disk io overhead - (say if you fsync
>> every 10 ms in batch (write latency in this case will be 10 ms as write
>> threads will be blocked for 10ms.  assuming continuous writes- you would be
>> issuing 1000/10 IO write IO to disk - 100 IOPS. If thats kept to 1ms to
>> reduce write latency to 1ms then IOPS becomes 1000)).
>>
>> So in case of batch mode it get tricky to balance latency & disk
>> utilisation. Testing this setting thoroughly on dev env would be
>> recommended as it can adversely affect performance. We had done some
>> benchmarks and had found 50ms to be ideal for our use case but thats
>> subjective as it leads to write latencies in excess of 50ms which could be
>> really high for some use cases. Though with modern day ssd's batch option
>> can be worthwhile to experiment with.
>>
>> A good description is also given here - http://stackoverflow.com/a/310
>> 33900/3646120
>>
>> On Thu, Apr 6, 2017 at 12:30 AM, Matija Gobec 
>> wrote:
>>
>>> Flushes have nothing to do with data persistence and node failure. Each
>>> write is acknowledged only when data has been written to the commit log AND
>>> memtable. That solves the issues of node failures and data consistency.
>>> When the node boots back up it replays commit log files and you don't loose
>>> data that was already written to that node.
>>>
>>> On Wed, Apr 5, 2017 at 6:22 PM, preetika tyagi 
>>> wrote:
>>>
 Hi,

 I read in Cassandra architecture documentation that if a node dies and
 there is some data in memtable which hasn't been written to the sstable,
 the commit log replay happens (assuming the commit log had been flushed to
 disk) when the node restarts and hence the data can be recovered.

 However, I was wondering if a node is fully dead for some reason with
 consistency level 1 (replication factor 3 but let's say it dies right after
 it finishes a request and hence the data hasn't been replicated to other
 nodes yet) and the data is only available in commit log on that node. Is
 there a way to recover data from this node which includes both sstable and
 commit log so that we can use it to replace it with a new node where we
 could replay commit log to recover the data?

 Thanks,
 Preetika

>>>
>>>
>>
>

Re: how to recover a dead node using commit log when memtable is lost

[preetika tyagi]

Very good explanation.
One follow-up question. If CL is set to 1 and RF to 3, then there are
chances of the data being lost if the machine crashes before replication
happens and the commit log (on the node which processed the data for CF=1)
is not synced yet. Right?

Thanks,
Preetika

On Wed, Apr 5, 2017 at 1:17 PM, Bhuvan Rawal  wrote:

> I beg to differ with @Matija here, IMO by default cassandra syncs data
> into commit log in a periodic fashion with a fsync period of 10 sec (Ref -
> https://github.com/apache/cassandra/blob/trunk/conf/cassandra.yaml#L361).
> If a write is not written to disk and RF is 1 else CL is Local One & node
> goes down then there could be potential data loss and client would expect
> data to be present.
>
> Therefore a good strategy would be to either have RF 3 and write with
> quorum. Or if thats not a feasible option then use Batch Mode for commitlog
> sync - That could lead to much higher disk io overhead - (say if you fsync
> every 10 ms in batch (write latency in this case will be 10 ms as write
> threads will be blocked for 10ms.  assuming continuous writes- you would be
> issuing 1000/10 IO write IO to disk - 100 IOPS. If thats kept to 1ms to
> reduce write latency to 1ms then IOPS becomes 1000)).
>
> So in case of batch mode it get tricky to balance latency & disk
> utilisation. Testing this setting thoroughly on dev env would be
> recommended as it can adversely affect performance. We had done some
> benchmarks and had found 50ms to be ideal for our use case but thats
> subjective as it leads to write latencies in excess of 50ms which could be
> really high for some use cases. Though with modern day ssd's batch option
> can be worthwhile to experiment with.
>
> A good description is also given here - http://stackoverflow.com/a/
> 31033900/3646120
>
> On Thu, Apr 6, 2017 at 12:30 AM, Matija Gobec 
> wrote:
>
>> Flushes have nothing to do with data persistence and node failure. Each
>> write is acknowledged only when data has been written to the commit log AND
>> memtable. That solves the issues of node failures and data consistency.
>> When the node boots back up it replays commit log files and you don't loose
>> data that was already written to that node.
>>
>> On Wed, Apr 5, 2017 at 6:22 PM, preetika tyagi 
>> wrote:
>>
>>> Hi,
>>>
>>> I read in Cassandra architecture documentation that if a node dies and
>>> there is some data in memtable which hasn't been written to the sstable,
>>> the commit log replay happens (assuming the commit log had been flushed to
>>> disk) when the node restarts and hence the data can be recovered.
>>>
>>> However, I was wondering if a node is fully dead for some reason with
>>> consistency level 1 (replication factor 3 but let's say it dies right after
>>> it finishes a request and hence the data hasn't been replicated to other
>>> nodes yet) and the data is only available in commit log on that node. Is
>>> there a way to recover data from this node which includes both sstable and
>>> commit log so that we can use it to replace it with a new node where we
>>> could replay commit log to recover the data?
>>>
>>> Thanks,
>>> Preetika
>>>
>>
>>
>

Re: Copy from CSV on OS X problem with varint values <= -2^63

[Boris Babic]

Stefania

This is the output of my --debug, I never touched CQLSH_NO_BUNDLED and did not 
know about it.
As you can see I have used homebrew to install Cassandra and looks like its the 
embedded version as it sits under the Cassandra folder ? 

cqlsh --debug
Using CQL driver: 
Using connect timeout: 5 seconds
Using 'utf-8' encoding
Using ssl: False
Connected to Test Cluster at 127.0.0.1:9042.
[cqlsh 5.0.1 | Cassandra 3.10 | CQL spec 3.4.4 | Native protocol v4]
Use HELP for help.

> On Apr 5, 2017, at 12:07 PM, Stefania Alborghetti 
>  wrote:
> 
> You are welcome.
> 
> I traced the problem to a commit of the Python driver that shipped in version 
> 3.8 of the driver. It is fixed in 3.8.1. More details on CASSANDRA-13408. I 
> don't think it's related to the OS.
> 
> Since Cassandra 3.10 ships with an older version of the driver embedded in a 
> zip file in the lib folder, and this version is not affected, I'm guessing 
> that either the embedded version does not work on OS X, or you are manually 
> using a different version of the driver by setting CQLSH_NO_BUNDLED (which is 
> why I could reproduce it on my laptop). 
> 
> You can run cqlsh with --debug to see the version of the driver that cqlsh is 
> using, for example:
> 
> cqlsh --debug
> Using CQL driver:  '/usr/local/lib/python2.7/dist-packages/cassandra_driver-3.8.1-py2.7-linux-x86_64.egg/cassandra/__init__.pyc'>
> 
> Can you confirm if you were overriding the Python driver by setting 
> CQLSH_NO_BUNDLED and the version of the driver?
> 
> 
> 
> On Tue, Apr 4, 2017 at 6:12 PM, Boris Babic  wrote:
> Thanks Stefania, going from memory don't think I noticed this on windows but 
> haven't got a machine handy to test it on at the moment. 
> 
> On Apr 4, 2017, at 19:44, Stefania Alborghetti 
>  wrote:
> 
>> I've reproduced the same problem on Linux, and I've opened CASSANDRA-13408. 
>> As a workaround, disable prepared statements and it will work (WITH HEADER = 
>> TRUE AND PREPAREDSTATEMENTS = False).
>> 
>> On Tue, Apr 4, 2017 at 5:02 PM, Boris Babic  wrote:
>> 
>> On Apr 4, 2017, at 7:00 PM, Boris Babic  wrote:
>> 
>> Hi
>> 
>> I’m testing the write of various datatypes on OS X for fun running cassandra 
>> 3.10 on a single laptop instance, and from what i can see varint should map 
>> to java.math.BigInteger and have no problems with Long.MIN_VALE , 
>> -9223372036854775808, but i can’t see what I’m doing wrong.
>> 
>> cqlsh: 5.0.1
>> cassandra 3.10
>> osx el capitan.
>> 
>> data.csv:
>> 
>> id,varint
>> -2147483648,-9223372036854775808
>> 2147483647,9223372036854775807
>> 
>> COPY mykeyspace.data (id,varint) FROM 'data.csv' WITH HEADER=true;
>> 
>>   Failed to make batch statement: Received an argument of invalid type 
>> for column "varint". Expected: , 
>> Got: ; (descriptor 'bit_length' requires a 'int' object but 
>> received a 'long’)
>> 
>> If I directly type a similar insert in cqlsh no such problem occurs, in fact 
>> I can make the value many orders of magnitude less and all is fine.
>> 
>> cqlsh> insert into mykeyspace.data (id,varint) 
>> values(1,-9223372036854775808898989898) ;
>> 
>> Had not observed this before on other OS, is this something todo with the 
>> way the copy from parser is interpreting varint for values <= -2^63 ?
>> 
>> Thanks for any input
>> Boris
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>> -- 
>> 
>> STEFANIA ALBORGHETTI
>> Software engineer | +852 6114 9265 | stefania.alborghe...@datastax.com
>> 
>> 
>> 
>> 
> 
> 
> 
> -- 
> 
> STEFANIA ALBORGHETTI
> Software engineer | +852 6114 9265 | stefania.alborghe...@datastax.com
> 
> 
> 
> 

Re: how to recover a dead node using commit log when memtable is lost

[Bhuvan Rawal]

I beg to differ with @Matija here, IMO by default cassandra syncs data into
commit log in a periodic fashion with a fsync period of 10 sec (Ref -
https://github.com/apache/cassandra/blob/trunk/conf/cassandra.yaml#L361).
If a write is not written to disk and RF is 1 else CL is Local One & node
goes down then there could be potential data loss and client would expect
data to be present.

Therefore a good strategy would be to either have RF 3 and write with
quorum. Or if thats not a feasible option then use Batch Mode for commitlog
sync - That could lead to much higher disk io overhead - (say if you fsync
every 10 ms in batch (write latency in this case will be 10 ms as write
threads will be blocked for 10ms.  assuming continuous writes- you would be
issuing 1000/10 IO write IO to disk - 100 IOPS. If thats kept to 1ms to
reduce write latency to 1ms then IOPS becomes 1000)).

So in case of batch mode it get tricky to balance latency & disk
utilisation. Testing this setting thoroughly on dev env would be
recommended as it can adversely affect performance. We had done some
benchmarks and had found 50ms to be ideal for our use case but thats
subjective as it leads to write latencies in excess of 50ms which could be
really high for some use cases. Though with modern day ssd's batch option
can be worthwhile to experiment with.

A good description is also given here -
http://stackoverflow.com/a/31033900/3646120

On Thu, Apr 6, 2017 at 12:30 AM, Matija Gobec  wrote:

> Flushes have nothing to do with data persistence and node failure. Each
> write is acknowledged only when data has been written to the commit log AND
> memtable. That solves the issues of node failures and data consistency.
> When the node boots back up it replays commit log files and you don't loose
> data that was already written to that node.
>
> On Wed, Apr 5, 2017 at 6:22 PM, preetika tyagi 
> wrote:
>
>> Hi,
>>
>> I read in Cassandra architecture documentation that if a node dies and
>> there is some data in memtable which hasn't been written to the sstable,
>> the commit log replay happens (assuming the commit log had been flushed to
>> disk) when the node restarts and hence the data can be recovered.
>>
>> However, I was wondering if a node is fully dead for some reason with
>> consistency level 1 (replication factor 3 but let's say it dies right after
>> it finishes a request and hence the data hasn't been replicated to other
>> nodes yet) and the data is only available in commit log on that node. Is
>> there a way to recover data from this node which includes both sstable and
>> commit log so that we can use it to replace it with a new node where we
>> could replay commit log to recover the data?
>>
>> Thanks,
>> Preetika
>>
>
>

Re: how to recover a dead node using commit log when memtable is lost

[Matija Gobec]

Flushes have nothing to do with data persistence and node failure. Each
write is acknowledged only when data has been written to the commit log AND
memtable. That solves the issues of node failures and data consistency.
When the node boots back up it replays commit log files and you don't loose
data that was already written to that node.

On Wed, Apr 5, 2017 at 6:22 PM, preetika tyagi 
wrote:

> Hi,
>
> I read in Cassandra architecture documentation that if a node dies and
> there is some data in memtable which hasn't been written to the sstable,
> the commit log replay happens (assuming the commit log had been flushed to
> disk) when the node restarts and hence the data can be recovered.
>
> However, I was wondering if a node is fully dead for some reason with
> consistency level 1 (replication factor 3 but let's say it dies right after
> it finishes a request and hence the data hasn't been replicated to other
> nodes yet) and the data is only available in commit log on that node. Is
> there a way to recover data from this node which includes both sstable and
> commit log so that we can use it to replace it with a new node where we
> could replay commit log to recover the data?
>
> Thanks,
> Preetika
>

Re: Making a Cassandra node cluster unique

[Matija Gobec]

You CAN have two separate clusters with same name and configuration.
Separation of the clusters is just a matter of defining seed nodes
properly. That being said, it doesn't mean you SHOULD have clusters with
same name.
We usually run same cluster name when testing on test/stage cluster and
deploying same hardware and configuration as a "production" cluster but
they are always in different physical networks and/or VPCs. Just make sure
nodes from one cluster can't see nodes from a seconds one and you are fine.

On Wed, Apr 5, 2017 at 3:15 PM, Hannu Kröger  wrote:

> Hi,
>
> Cluster name should be unique because with misconfiguration you might make
> the nodes connect to either of the cluster and then you will have nodes is
> wrong clusters.
>
> Theoretically it can work with same names as well but to be on the safe
> side, make the cluster names unique.
>
> Hannu
>
> On Wed, 5 Apr 2017 at 8.36, William Boutin 
> wrote:
>
>> Someone on my team asked me a question that I could not find an easy
>> answer and I was hoping someone could answer for me.
>>
>> When we configure Cassandra, we use the Cluster Name, Data Center, and
>> Rack to define the group of Cassandra nodes involved in holding our
>> keyspace records. If a second set of nodes had the same Cluster Name, Data
>> Center, and Rack values, is there a chance that CRUD actions directed at
>> the first cluster of nodes could somehow end up at the second cluster of
>> nodes?
>>
>> Thank you in advance.
>>
>>
>>
>>
>>
>> [image: Ericsson] 
>>
>> *WILLIAM L. BOUTIN *
>> Engineer IV - Sftwr
>> BMDA PADB DSE DU CC NGEE
>>
>>
>> *Ericsson*
>> 1 Ericsson Drive, US PI06 1.S747
>> Piscataway, NJ, 08854, USA
>> Phone (913) 241-5574
>> Mobile (732) 213-1368
>> Emergency (732) 354-1263
>> william.bou...@ericsson.com
>> www.ericsson.com
>>
>> [image: http://www.ericsson.com/current_campaign]
>> 
>>
>> Legal entity: EUS - ERICSSON INC., registered office in US PI01 4A242.
>> This Communication is Confidential. We only send and receive email on the
>> basis of the terms set out at www.ericsson.com/email_disclaimer
>>
>>
>>
>

Re: Cassandra and LINUX CPU Context Switches

[daemeon reiydelle]

This would be normal if the switches are user to kernel mode (disk &
network IO are kernel mode activities). If your run queue (jobs waiting to
run) is much larger than the number of cores (just a swag but less than
2-3*# of cores), you might have other issues.


*...*



*Daemeon C.M. ReiydelleUSA (+1) 415.501.0198London (+44) (0) 20 8144 9872*

On Wed, Apr 5, 2017 at 5:45 AM, William Boutin 
wrote:

> I’ve noticed that my apache-cassandra 2.2.6 process is consistently
> performing CPU Context Switches above 10,000 per second.
>
> Is this to be expected or should I be looking into ways to lower the
> number of context switches done on my Cassandra cluster?
>
> Thanks in advance.
>
>
>
> [image: Ericsson] 
>
> *WILLIAM L. BOUTIN *
> Engineer IV - Sftwr
> BMDA PADB DSE DU CC NGEE
>
>
> *Ericsson*
> 1 Ericsson Drive, US PI06 1.S747
> Piscataway, NJ, 08854, USA
> Phone (913) 241-5574
> Mobile (732) 213-1368
> Emergency (732) 354-1263
> william.bou...@ericsson.com
> www.ericsson.com
>
> [image: http://www.ericsson.com/current_campaign]
> 
>
> Legal entity: EUS - ERICSSON INC., registered office in US PI01 4A242.
> This Communication is Confidential. We only send and receive email on the
> basis of the terms set out at www.ericsson.com/email_disclaimer
>
>
>

how to recover a dead node using commit log when memtable is lost

[preetika tyagi]

Hi,

I read in Cassandra architecture documentation that if a node dies and
there is some data in memtable which hasn't been written to the sstable,
the commit log replay happens (assuming the commit log had been flushed to
disk) when the node restarts and hence the data can be recovered.

However, I was wondering if a node is fully dead for some reason with
consistency level 1 (replication factor 3 but let's say it dies right after
it finishes a request and hence the data hasn't been replicated to other
nodes yet) and the data is only available in commit log on that node. Is
there a way to recover data from this node which includes both sstable and
commit log so that we can use it to replace it with a new node where we
could replay commit log to recover the data?

Thanks,
Preetika

antlr-runtime-3.2.jar is turning into 0 bytes and dse is going down

[Surbhi Gupta]

Hi,

We have single node instance where we have cassandra , mysql and
application running at the same node for developers.

We are at dse 4.8.9 and dse is going down after sometime .
What we have noticed is that few of the jar at /usr/share/dse/common are
turning into 0 bytes.
Jars are as follows:

antlr-runtime-3.2.jar
antlr-2.7.7.jar
antlr-3.2.jar


When i install the dse all jars are of non zero bytes but not sure what is
happening and some of the jars are turning into 0 bytes and dse is shutting
down.

We were earlier at dse 4.6.10 and we did not see this issue.

Anybody faced this?
Any suggestion?

Thanks
Surbhi

Re: Effective partition key for time series data, which allows range queries?

[Jim Ancona]

That's an interesting refinement! I'll keep it in mind the next time this
sort of thing comes up.

Jim

On Wed, Apr 5, 2017 at 9:22 AM, Eric Stevens  wrote:

> Jim's basic model is similar to how we've solved this exact kind of
> problem many times.  From my own experience, I strongly recommend that you
> make a `bucket` field in the partition key, and a `time` field in the
> cluster key.  Make both of these of data type `timestamp`.  Then use
> application logic to floor the bucket to an appropriate interval according
> to your chosen bucket size.
>
> The reason is that as your data needs grow, the one thing you can be
> pretty confident in is that your assumptions about data density per
> partition will turn out to be eventually wrong.  This is either because of
> expanding requirements (you're adding new fields to this table), because of
> increased application usage (you're being successful!), or because you
> didn't anticipate a use case with different data density per bucket than
> you anticipated (you're not prescient).
>
> It's easy in application code to adjust your timestamp interval if your
> keying allows for arbitrary adjustments.  Most often you're going to end up
> making smaller buckets as your needs progress.  The upshot is that with a
> little careful selection of bucketing strategy, partition key range
> iterations are still correct if you adjust from say a 24 hour bucket to a
> 12 hour, 6 hour, 3 hour, 1 hour, 30 minute, 15 minute, or 1 minute bucket.
> The data written under the larger bucket size still lands on a smaller
> bucket interval, so you don't really even need to use complex logic in the
> application to adapt to the new bucket size.  You definitely don't want to
> paint yourself into a corner where you need a smaller bucket size but your
> data model didn't leave room for it.
>
> On Tue, Apr 4, 2017 at 2:59 PM Jim Ancona  wrote:
>
>> The typical recommendation for maximum partition size is on the order of
>> 100mb and/or 100,000 rows. That's not a hard limit, but you may be setting
>> yourself up for issues as you approach or exceed those numbers.
>>
>> If you need to reduce partition size, the typical way to do this is by
>> "bucketing," that is adding a synthetic column to the partition key to
>> separate the data into separate buckets. In your example above, I assume
>> that your current primary key is (user, timestamp), where user is the
>> partition key and timestamp is the clustering key. Say that you want to
>> store a maximum of a years worth of data in a partition. You would make the
>> primary key be ((user, year), timestamp). The partition key is now
>> (user, year) where year is the year part of the timestamp. Now if you want
>> to query the data for last month, you would do:
>>
>> select * from data where user_id = 'foo' and year = 2017 and timestamp >=
>> '<1 month ago>' and timestamp <= ''
>>
>>
>> If you wanted to get the data for the last 6 month, you'd do something
>> like:
>>
>> select * from data where user_id = 'foo' and year in (2016, 2017) and
>> timestamp >= '<6 months ago>' and timestamp <= ''  (Notice that
>> because the query spans two years, you have to include both years in the
>> select criteria so that C* knows which partitions to query. )
>>
>>
>> You can make the buckets smaller (e.g. weeks, days, hours instead of
>> years), but of course querying multiple buckets is less efficient, so it's
>> worth making your buckets as large as you can without making them too big.
>>
>> Hope this helps!
>>
>> Jim
>>
>>
>>
>>
>> On Mon, Mar 27, 2017 at 8:47 PM, Ali Akhtar  wrote:
>>
>> I have a use case where the data for individual users is being tracked,
>> and every 15 minutes or so, the data for the past 15 minutes is inserted
>> into the table.
>>
>> The table schema looks like:
>> user id, timestamp, foo, bar, etc.
>>
>> Where foo, bar, etc are the items being tracked, and their values over
>> the past 15 minutes.
>>
>> I initially planned to use the user id as the primary key of the table.
>> But, I realized that this may cause really wide rows ( tracking for 24
>> hours means 96 records inserted (1 for each 15 min window), over 1 year
>> this means 36k records per user, over 2 years, 72k, etc).
>>
>> I know the  limit of wide rows is billions of records, but I've heard
>> that the practical limit is much lower.
>>
>> So I considered using a composite primary key: (user, timestamp)
>>
>> If I'm correct, the above should create a new row for each user &
>> timestamp logged.
>>
>> However, will i still be able to do range queries on the timestamp, to
>> e.g return the data for the last week?
>>
>> E.g select * from data where user_id = 'foo' and timestamp >= '<1 month
>> ago>' and timestamp <= '' ?
>>
>>
>>

Re: Effective partition key for time series data, which allows range queries?

[Eric Stevens]

Jim's basic model is similar to how we've solved this exact kind of problem
many times.  From my own experience, I strongly recommend that you make a
`bucket` field in the partition key, and a `time` field in the cluster
key.  Make both of these of data type `timestamp`.  Then use application
logic to floor the bucket to an appropriate interval according to your
chosen bucket size.

The reason is that as your data needs grow, the one thing you can be pretty
confident in is that your assumptions about data density per partition will
turn out to be eventually wrong.  This is either because of expanding
requirements (you're adding new fields to this table), because of increased
application usage (you're being successful!), or because you didn't
anticipate a use case with different data density per bucket than you
anticipated (you're not prescient).

It's easy in application code to adjust your timestamp interval if your
keying allows for arbitrary adjustments.  Most often you're going to end up
making smaller buckets as your needs progress.  The upshot is that with a
little careful selection of bucketing strategy, partition key range
iterations are still correct if you adjust from say a 24 hour bucket to a
12 hour, 6 hour, 3 hour, 1 hour, 30 minute, 15 minute, or 1 minute bucket.
The data written under the larger bucket size still lands on a smaller
bucket interval, so you don't really even need to use complex logic in the
application to adapt to the new bucket size.  You definitely don't want to
paint yourself into a corner where you need a smaller bucket size but your
data model didn't leave room for it.

On Tue, Apr 4, 2017 at 2:59 PM Jim Ancona  wrote:

> The typical recommendation for maximum partition size is on the order of
> 100mb and/or 100,000 rows. That's not a hard limit, but you may be setting
> yourself up for issues as you approach or exceed those numbers.
>
> If you need to reduce partition size, the typical way to do this is by
> "bucketing," that is adding a synthetic column to the partition key to
> separate the data into separate buckets. In your example above, I assume
> that your current primary key is (user, timestamp), where user is the
> partition key and timestamp is the clustering key. Say that you want to
> store a maximum of a years worth of data in a partition. You would make the
> primary key be ((user, year), timestamp). The partition key is now (user,
> year) where year is the year part of the timestamp. Now if you want to
> query the data for last month, you would do:
>
> select * from data where user_id = 'foo' and year = 2017 and timestamp >=
> '<1 month ago>' and timestamp <= ''
>
>
> If you wanted to get the data for the last 6 month, you'd do something
> like:
>
> select * from data where user_id = 'foo' and year in (2016, 2017) and
> timestamp >= '<6 months ago>' and timestamp <= ''  (Notice that
> because the query spans two years, you have to include both years in the
> select criteria so that C* knows which partitions to query. )
>
>
> You can make the buckets smaller (e.g. weeks, days, hours instead of
> years), but of course querying multiple buckets is less efficient, so it's
> worth making your buckets as large as you can without making them too big.
>
> Hope this helps!
>
> Jim
>
>
>
>
> On Mon, Mar 27, 2017 at 8:47 PM, Ali Akhtar  wrote:
>
> I have a use case where the data for individual users is being tracked,
> and every 15 minutes or so, the data for the past 15 minutes is inserted
> into the table.
>
> The table schema looks like:
> user id, timestamp, foo, bar, etc.
>
> Where foo, bar, etc are the items being tracked, and their values over the
> past 15 minutes.
>
> I initially planned to use the user id as the primary key of the table.
> But, I realized that this may cause really wide rows ( tracking for 24
> hours means 96 records inserted (1 for each 15 min window), over 1 year
> this means 36k records per user, over 2 years, 72k, etc).
>
> I know the  limit of wide rows is billions of records, but I've heard that
> the practical limit is much lower.
>
> So I considered using a composite primary key: (user, timestamp)
>
> If I'm correct, the above should create a new row for each user &
> timestamp logged.
>
> However, will i still be able to do range queries on the timestamp, to e.g
> return the data for the last week?
>
> E.g select * from data where user_id = 'foo' and timestamp >= '<1 month
> ago>' and timestamp <= '' ?
>
>
>

Re: Making a Cassandra node cluster unique

[Hannu Kröger]

Hi,

Cluster name should be unique because with misconfiguration you might make
the nodes connect to either of the cluster and then you will have nodes is
wrong clusters.

Theoretically it can work with same names as well but to be on the safe
side, make the cluster names unique.

Hannu
On Wed, 5 Apr 2017 at 8.36, William Boutin 
wrote:

> Someone on my team asked me a question that I could not find an easy
> answer and I was hoping someone could answer for me.
>
> When we configure Cassandra, we use the Cluster Name, Data Center, and
> Rack to define the group of Cassandra nodes involved in holding our
> keyspace records. If a second set of nodes had the same Cluster Name, Data
> Center, and Rack values, is there a chance that CRUD actions directed at
> the first cluster of nodes could somehow end up at the second cluster of
> nodes?
>
> Thank you in advance.
>
>
>
>
>
> [image: Ericsson] 
>
> *WILLIAM L. BOUTIN *
> Engineer IV - Sftwr
> BMDA PADB DSE DU CC NGEE
>
>
> *Ericsson*
> 1 Ericsson Drive, US PI06 1.S747
> Piscataway, NJ, 08854, USA
> Phone (913) 241-5574
> Mobile (732) 213-1368
> Emergency (732) 354-1263
> william.bou...@ericsson.com
> www.ericsson.com
>
> [image: http://www.ericsson.com/current_campaign]
> 
>
> Legal entity: EUS - ERICSSON INC., registered office in US PI01 4A242.
> This Communication is Confidential. We only send and receive email on the
> basis of the terms set out at www.ericsson.com/email_disclaimer
>
>
>

Cassandra and LINUX CPU Context Switches

[William Boutin]

I've noticed that my apache-cassandra 2.2.6 process is consistently performing 
CPU Context Switches above 10,000 per second.
Is this to be expected or should I be looking into ways to lower the number of 
context switches done on my Cassandra cluster?
Thanks in advance.

[Ericsson]

WILLIAM L. BOUTIN
Engineer IV - Sftwr
BMDA PADB DSE DU CC NGEE

Ericsson
1 Ericsson Drive, US PI06 1.S747
Piscataway, NJ, 08854, USA
Phone (913) 241-5574
Mobile (732) 213-1368
Emergency (732) 354-1263
william.bou...@ericsson.com
www.ericsson.com
[http://www.ericsson.com/current_campaign]
Legal entity: EUS - ERICSSON INC., registered office in US PI01 4A242. This 
Communication is Confidential. We only send and receive email on the basis of 
the terms set out at 
www.ericsson.com/email_disclaimer

Making a Cassandra node cluster unique

[William Boutin]

Someone on my team asked me a question that I could not find an easy answer and 
I was hoping someone could answer for me.
When we configure Cassandra, we use the Cluster Name, Data Center, and Rack to 
define the group of Cassandra nodes involved in holding our keyspace records. 
If a second set of nodes had the same Cluster Name, Data Center, and Rack 
values, is there a chance that CRUD actions directed at the first cluster of 
nodes could somehow end up at the second cluster of nodes?
Thank you in advance.


[Ericsson]

WILLIAM L. BOUTIN
Engineer IV - Sftwr
BMDA PADB DSE DU CC NGEE

Ericsson
1 Ericsson Drive, US PI06 1.S747
Piscataway, NJ, 08854, USA
Phone (913) 241-5574
Mobile (732) 213-1368
Emergency (732) 354-1263
william.bou...@ericsson.com
www.ericsson.com
[http://www.ericsson.com/current_campaign]
Legal entity: EUS - ERICSSON INC., registered office in US PI01 4A242. This 
Communication is Confidential. We only send and receive email on the basis of 
the terms set out at 
www.ericsson.com/email_disclaimer