RE: Reading SSTables Potential File Descriptor Leak 1.2.18

[Job Thomas]

Hi, It look like the offset in keycache is wrong !!.
 
 refresh the keycache may solve the issue.
 
Thanks & Regards
Job M Thomas 
Platform & Technology



From: Tim Heckman [mailto:t...@pagerduty.com]
Sent: Wed 9/24/2014 6:17 AM
To: user@cassandra.apache.org
Subject: Reading SSTables Potential File Descriptor Leak 1.2.18


Hello,

I ran in to a problem today where Cassandra 1.2.18 exhausted its number of 
permitted open file descriptors (65,535). This node has 256 tokens (vnodes) and 
runs in a test environment with relatively little traffic/data.

As best I could tell, the majority of the file descriptors open were for a 
single SSTable '.db' file. Looking in the error logs I found quite a few 
exceptions that looked to have been identical: 

ERROR [ReadStage:3817] 2014-09-19 07:00:11,056 CassandraDaemon.java (line 191) 
Exception in thread Thread[ReadStage:3817,5,main]
java.lang.RuntimeException: java.lang.IllegalArgumentException: unable to seek 
to position 29049 in /mnt/var/lib/cassandra/data/path/to/file.db (1855 bytes) 
in read-only mode

Upon further investigation, it turns out this file became 'read-only' after the 
Cassandra node was gracefully restarted last week. I'd imagine this is a 
discussion for another email thread.

I fixed the issue by running:

nodetool scrub 
nodetool repair 

Attached to this email is one of the log entries/stacktrace for this exception.

Before opening a JIRA ticket I thought I'd reach out to the list to see if 
anyone has seen any similar behavior as well as do a bit of source-diving to 
try and verify that the descriptor is actually leaking.

Cheers!
-Tim

Reading SSTables Potential File Descriptor Leak 1.2.18

[Tim Heckman]

Hello,

I ran in to a problem today where Cassandra 1.2.18 exhausted its number of
permitted open file descriptors (65,535). This node has 256 tokens (vnodes)
and runs in a test environment with relatively little traffic/data.

As best I could tell, the majority of the file descriptors open were for a
single SSTable '.db' file. Looking in the error logs I found quite a few
exceptions that looked to have been identical:

ERROR [ReadStage:3817] 2014-09-19 07:00:11,056 CassandraDaemon.java (line
191) Exception in thread Thread[ReadStage:3817,5,main]
java.lang.RuntimeException: java.lang.IllegalArgumentException: unable to
seek to position 29049 in /mnt/var/lib/cassandra/data/path/to/file.db (1855
bytes) in read-only mode

Upon further investigation, it turns out this file became 'read-only' after
the Cassandra node was gracefully restarted last week. I'd imagine this is
a discussion for another email thread.

I fixed the issue by running:

nodetool scrub 
nodetool repair 

Attached to this email is one of the log entries/stacktrace for this
exception.

Before opening a JIRA ticket I thought I'd reach out to the list to see if
anyone has seen any similar behavior as well as do a bit of source-diving
to try and verify that the descriptor is actually leaking.

Cheers!
-Tim
ERROR [ReadStage:3817] 2014-09-19 07:00:11,056 CassandraDaemon.java (line 191) 
Exception in thread Thread[ReadStage:3817,5,main]
java.lang.RuntimeException: java.lang.IllegalArgumentException: unable to seek 
to position 29049 in 
/mnt/var/lib/cassandra/data/IzanagiQueue/WorkQueue/IzanagiQueue-WorkQueue-ic-1-Data.db
 (1855 bytes) in read-only mode
at 
org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1626)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.lang.IllegalArgumentException: unable to seek to position 29049 
in 
/mnt/var/lib/cassandra/data/IzanagiQueue/WorkQueue/IzanagiQueue-WorkQueue-ic-1-Data.db
 (1855 bytes) in read-only mode
at 
org.apache.cassandra.io.util.RandomAccessReader.seek(RandomAccessReader.java:306)
at 
org.apache.cassandra.io.util.PoolingSegmentedFile.getSegment(PoolingSegmentedFile.java:42)
at 
org.apache.cassandra.io.sstable.SSTableReader.getFileDataInput(SSTableReader.java:1048)
at 
org.apache.cassandra.db.columniterator.IndexedSliceReader.setToRowStart(IndexedSliceReader.java:130)
at 
org.apache.cassandra.db.columniterator.IndexedSliceReader.(IndexedSliceReader.java:91)
at 
org.apache.cassandra.db.columniterator.SSTableSliceIterator.createReader(SSTableSliceIterator.java:68)
at 
org.apache.cassandra.db.columniterator.SSTableSliceIterator.(SSTableSliceIterator.java:44)
at 
org.apache.cassandra.db.filter.SliceQueryFilter.getSSTableColumnIterator(SliceQueryFilter.java:104)
at 
org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFilter.java:68)
at 
org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:272)
at 
org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:65)
at 
org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1398)
at 
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1214)
at 
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1130)
at org.apache.cassandra.db.Table.getRow(Table.java:348)
at 
org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:70)
at 
org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:1070)
at 
org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1622)
... 3 more

Re: How to get data which has changed within x minutes using CQL?

[DuyHai Doan]

No, you need to compute yourself now - 15mins. CQL3 does not offer built-in
functions to deal with dates right now
 Le 24 sept. 2014 00:47, "Check Peck"  a écrit :

>
> On Tue, Sep 23, 2014 at 3:41 PM, DuyHai Doan  wrote:
>
>> now - 15 mins
>
>
>
> Can I run like this in CQL using cqlsh?
>
> SELECT * FROM client_data WHERE client_id = 1 and last_modified_date >=
> now - 15 mins
>
> When I ran the above query I got an error on my cql client -
>
> Bad Request: line 1:81 no viable alternative at input '-'
>
>

Re: How to get data which has changed within x minutes using CQL?

[Check Peck]

On Tue, Sep 23, 2014 at 3:41 PM, DuyHai Doan  wrote:

> now - 15 mins



Can I run like this in CQL using cqlsh?

SELECT * FROM client_data WHERE client_id = 1 and last_modified_date >= now
- 15 mins

When I ran the above query I got an error on my cql client -

Bad Request: line 1:81 no viable alternative at input '-'

Re: How to get data which has changed within x minutes using CQL?

[DuyHai Doan]

let previous15Min = now - 15 mins

SELECT * FROM client_data WHERE client_id = 1 and last_modified_date >=
previous15Min

Same thing for last 5 mins

On Wed, Sep 24, 2014 at 12:32 AM, Check Peck 
wrote:

> Yes I can provide client_id in my where clause. So now my query pattern
> will be -
>
> Give me everything for what has changed within last 15 minutes or 5
> minutes whose client_id is equal to 1?
>
> How does my query will look like then?
>
>
> On Tue, Sep 23, 2014 at 3:26 PM, DuyHai Doan  wrote:
>
>> It is possible to request a "range" of data according to the
>> last_modified_date but you still need to provide the client_id , the
>> partition key, in any case
>>
>>
>> On Wed, Sep 24, 2014 at 12:23 AM, Check Peck 
>> wrote:
>>
>>> I have a table structure like below -
>>>
>>> CREATE TABLE client_data (
>>>   client_id int,
>>>   consumer_id text,
>>>   last_modified_date timestamp,
>>>   PRIMARY KEY (client_id, last_modified_date, consumer_id)
>>> )
>>>
>>> I have a query pattern like this - Give me everything for what has
>>> changed withing last 15 minutes or 5 minutes? Is this possible to in CQL
>>> with the above tables?
>>>
>>
>>
>

Re: How to get data which has changed within x minutes using CQL?

[Check Peck]

Yes I can provide client_id in my where clause. So now my query pattern
will be -

Give me everything for what has changed within last 15 minutes or 5 minutes
whose client_id is equal to 1?

How does my query will look like then?


On Tue, Sep 23, 2014 at 3:26 PM, DuyHai Doan  wrote:

> It is possible to request a "range" of data according to the
> last_modified_date but you still need to provide the client_id , the
> partition key, in any case
>
>
> On Wed, Sep 24, 2014 at 12:23 AM, Check Peck 
> wrote:
>
>> I have a table structure like below -
>>
>> CREATE TABLE client_data (
>>   client_id int,
>>   consumer_id text,
>>   last_modified_date timestamp,
>>   PRIMARY KEY (client_id, last_modified_date, consumer_id)
>> )
>>
>> I have a query pattern like this - Give me everything for what has
>> changed withing last 15 minutes or 5 minutes? Is this possible to in CQL
>> with the above tables?
>>
>
>

Re: How to get data which has changed within x minutes using CQL?

[DuyHai Doan]

It is possible to request a "range" of data according to the
last_modified_date but you still need to provide the client_id , the
partition key, in any case


On Wed, Sep 24, 2014 at 12:23 AM, Check Peck 
wrote:

> I have a table structure like below -
>
> CREATE TABLE client_data (
>   client_id int,
>   consumer_id text,
>   last_modified_date timestamp,
>   PRIMARY KEY (client_id, last_modified_date, consumer_id)
> )
>
> I have a query pattern like this - Give me everything for what has changed
> withing last 15 minutes or 5 minutes? Is this possible to in CQL with the
> above tables?
>

How to get data which has changed within x minutes using CQL?

[Check Peck]

I have a table structure like below -

CREATE TABLE client_data (
  client_id int,
  consumer_id text,
  last_modified_date timestamp,
  PRIMARY KEY (client_id, last_modified_date, consumer_id)
)

I have a query pattern like this - Give me everything for what has changed
withing last 15 minutes or 5 minutes? Is this possible to in CQL with the
above tables?

Re: CPU consumption of Cassandra

[DuyHai Doan]

Nice catch Daniel. The comment from Sylvain explains a lot !

On Tue, Sep 23, 2014 at 11:33 PM, Daniel Chia  wrote:

> If I had to guess, it might be in part i could be due to inefficiencies in
> 2.0 with regards to CompositeType (which is used in CQL3 tables) -
> https://issues.apache.org/jira/browse/CASSANDRA-5417?focusedCommentId=13821243&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13821243
>
> Ticket reports 45% performance increase in reading slices compared to
> trunk in 2.1
>
> Thanks,
> Daniel
>
> On Tue, Sep 23, 2014 at 5:08 PM, DuyHai Doan  wrote:
>
>> I had done some benching in the past when we faced high CPU usage even
>> though data set is very small, sitting entirely in memory, read the report
>> there: https://github.com/doanduyhai/Cassandra_Data_Model_Bench
>>
>>  Our *partial *conclusion were:
>>
>>  1) slice query fetching a page of 64kb of data and decoding columns is
>> more CPU-expensive than a single read by column
>>  2) the decoding of CompositeType costs more CPU for CQL3 data model than
>> for old Thrift column family
>>  3) since the Cell type for all CQL3 table is forced to BytesType to
>> support any type of data, serialization/de-serialization may have a cost on
>> CPU.
>>
>> The issue Eric Leleu is facing reminds me of point 1). When he puts limit
>> to 1, it's a single read by column. The other query with limit 10 is
>> translated internally to a slice query and may explain the CPU difference
>>
>>  Now, do not take my words as granted. Those points are just *assumptions
>> *and partial conclusions. I need extensive in depth debugging to confirm
>> those. Did not have time lately.
>>
>> On Tue, Sep 23, 2014 at 10:46 PM, Chris Lohfink > > wrote:
>>
>>> CPU consumption may be affected from the cassandra-stress tool in 2nd
>>> example as well.  Running on a separate system eliminates it as a possible
>>> cause.  There is a little extra work but not anything that I think would be
>>> that obvious.  tracing (can enable with nodetool) or profiling (ie with
>>> yourkit) can give more exposure to the bottleneck.  Id run test from
>>> separate system first.
>>>
>>> ---
>>> Chris Lohfink
>>>
>>>
>>> On Sep 23, 2014, at 12:48 PM, Leleu Eric 
>>> wrote:
>>>
>>> First of all, Thanks for your help ! :)
>>>
>>> Here is some details :
>>>
>>> With RF=N=2 your essentially testing a single machine locally which isnt
>>> the best indicator long term
>>>
>>> I will  test with more nodes, (4 with RF = 2) but for now I'm limited to
>>> 2 nodes for non technical reason ...
>>>
>>> Well, first off you shouldn't run stress tool on the node your testing.
>>> Give it its own box.
>>>
>>> I performed the test in a new Keyspace in order to have a clear dataset.
>>>
>>> the 2nd query since its returning 10x the data and there will be more to
>>> go through within the partition
>>>
>>> I configured cassandra-stress in a way of each user has only one bucket
>>> so the amount of data is the same in the both case. ("select * from buckets
>>> where name = ? and tenantid = ? limit 1" and "select * from
>>> owner_to_buckets  where owner = ? and tenantid = ? limit 10").
>>> Does cassandra perform extra read when the limit is bigger than the
>>> available data (even if the partition key contains only one single value in
>>> the clustering column) ?
>>> If the amount of data is the same, how can we explain the difference of
>>> CPU consumption?
>>>
>>>
>>> Regards,
>>> Eric
>>>
>>> 
>>> De : Chris Lohfink [clohf...@blackbirdit.com]
>>> Date d'envoi : mardi 23 septembre 2014 19:23
>>> À : user@cassandra.apache.org
>>> Objet : Re: CPU consumption of Cassandra
>>>
>>> Well, first off you shouldn't run stress tool on the node your testing.
>>> Give it its own box.
>>>
>>> With RF=N=2 your essentially testing a single machine locally which isnt
>>> the best indicator long term (optimizations available when reading data
>>> thats local to the node).  80k/sec on a system is pretty good though, your
>>> probably seeing slower on the 2nd query since its returning 10x the data
>>> and there will be more to go through within the partition. 42k/sec is still
>>> acceptable imho since these are smaller boxes.  You are probably seeing
>>> high CPU because the system is doing a lot :)
>>>
>>> If you want to get more out of these systems can do some tuning
>>> probably, enable trace to see whats actually the bottleneck.
>>>
>>> Collections will very likely hurt more then help.
>>>
>>> ---
>>> Chris Lohfink
>>>
>>> On Sep 23, 2014, at 9:39 AM, Leleu Eric >> mailto:eric.le...@worldline.com >> wrote:
>>>
>>> I tried to run “cassandra-stress” on some of my table as proposed by
>>> Jake Luciani.
>>>
>>> For a simple table, this tool is able to perform 8 read op/s with a
>>> few CPU consumption if I request the table by the PK(name, tenanted)
>>>
>>> Ex :
>>> TABLE :
>>>
>>> CREATE TABLE IF NOT EXISTS buckets (tenantid varchar,
>>> name varchar,
>>> owner var

Re: CPU consumption of Cassandra

[Daniel Chia]

If I had to guess, it might be in part i could be due to inefficiencies in
2.0 with regards to CompositeType (which is used in CQL3 tables) -
https://issues.apache.org/jira/browse/CASSANDRA-5417?focusedCommentId=13821243&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13821243

Ticket reports 45% performance increase in reading slices compared to trunk
in 2.1

Thanks,
Daniel

On Tue, Sep 23, 2014 at 5:08 PM, DuyHai Doan  wrote:

> I had done some benching in the past when we faced high CPU usage even
> though data set is very small, sitting entirely in memory, read the report
> there: https://github.com/doanduyhai/Cassandra_Data_Model_Bench
>
>  Our *partial *conclusion were:
>
>  1) slice query fetching a page of 64kb of data and decoding columns is
> more CPU-expensive than a single read by column
>  2) the decoding of CompositeType costs more CPU for CQL3 data model than
> for old Thrift column family
>  3) since the Cell type for all CQL3 table is forced to BytesType to
> support any type of data, serialization/de-serialization may have a cost on
> CPU.
>
> The issue Eric Leleu is facing reminds me of point 1). When he puts limit
> to 1, it's a single read by column. The other query with limit 10 is
> translated internally to a slice query and may explain the CPU difference
>
>  Now, do not take my words as granted. Those points are just *assumptions
> *and partial conclusions. I need extensive in depth debugging to confirm
> those. Did not have time lately.
>
> On Tue, Sep 23, 2014 at 10:46 PM, Chris Lohfink 
> wrote:
>
>> CPU consumption may be affected from the cassandra-stress tool in 2nd
>> example as well.  Running on a separate system eliminates it as a possible
>> cause.  There is a little extra work but not anything that I think would be
>> that obvious.  tracing (can enable with nodetool) or profiling (ie with
>> yourkit) can give more exposure to the bottleneck.  Id run test from
>> separate system first.
>>
>> ---
>> Chris Lohfink
>>
>>
>> On Sep 23, 2014, at 12:48 PM, Leleu Eric 
>> wrote:
>>
>> First of all, Thanks for your help ! :)
>>
>> Here is some details :
>>
>> With RF=N=2 your essentially testing a single machine locally which isnt
>> the best indicator long term
>>
>> I will  test with more nodes, (4 with RF = 2) but for now I'm limited to
>> 2 nodes for non technical reason ...
>>
>> Well, first off you shouldn't run stress tool on the node your testing.
>> Give it its own box.
>>
>> I performed the test in a new Keyspace in order to have a clear dataset.
>>
>> the 2nd query since its returning 10x the data and there will be more to
>> go through within the partition
>>
>> I configured cassandra-stress in a way of each user has only one bucket
>> so the amount of data is the same in the both case. ("select * from buckets
>> where name = ? and tenantid = ? limit 1" and "select * from
>> owner_to_buckets  where owner = ? and tenantid = ? limit 10").
>> Does cassandra perform extra read when the limit is bigger than the
>> available data (even if the partition key contains only one single value in
>> the clustering column) ?
>> If the amount of data is the same, how can we explain the difference of
>> CPU consumption?
>>
>>
>> Regards,
>> Eric
>>
>> 
>> De : Chris Lohfink [clohf...@blackbirdit.com]
>> Date d'envoi : mardi 23 septembre 2014 19:23
>> À : user@cassandra.apache.org
>> Objet : Re: CPU consumption of Cassandra
>>
>> Well, first off you shouldn't run stress tool on the node your testing.
>> Give it its own box.
>>
>> With RF=N=2 your essentially testing a single machine locally which isnt
>> the best indicator long term (optimizations available when reading data
>> thats local to the node).  80k/sec on a system is pretty good though, your
>> probably seeing slower on the 2nd query since its returning 10x the data
>> and there will be more to go through within the partition. 42k/sec is still
>> acceptable imho since these are smaller boxes.  You are probably seeing
>> high CPU because the system is doing a lot :)
>>
>> If you want to get more out of these systems can do some tuning probably,
>> enable trace to see whats actually the bottleneck.
>>
>> Collections will very likely hurt more then help.
>>
>> ---
>> Chris Lohfink
>>
>> On Sep 23, 2014, at 9:39 AM, Leleu Eric > mailto:eric.le...@worldline.com >> wrote:
>>
>> I tried to run “cassandra-stress” on some of my table as proposed by Jake
>> Luciani.
>>
>> For a simple table, this tool is able to perform 8 read op/s with a
>> few CPU consumption if I request the table by the PK(name, tenanted)
>>
>> Ex :
>> TABLE :
>>
>> CREATE TABLE IF NOT EXISTS buckets (tenantid varchar,
>> name varchar,
>> owner varchar,
>> location varchar,
>> description varchar,
>> codeQuota varchar,
>> creationDate timestamp,
>> updateDate timestamp,
>> PRIMARY KEY (name, tenantid));
>>
>> QUERY : select * from buckets where name = ? and tenantid = ? limit 1;
>>
>> TOP ou

Re: CPU consumption of Cassandra

[DuyHai Doan]

I had done some benching in the past when we faced high CPU usage even
though data set is very small, sitting entirely in memory, read the report
there: https://github.com/doanduyhai/Cassandra_Data_Model_Bench

 Our *partial *conclusion were:

 1) slice query fetching a page of 64kb of data and decoding columns is
more CPU-expensive than a single read by column
 2) the decoding of CompositeType costs more CPU for CQL3 data model than
for old Thrift column family
 3) since the Cell type for all CQL3 table is forced to BytesType to
support any type of data, serialization/de-serialization may have a cost on
CPU.

The issue Eric Leleu is facing reminds me of point 1). When he puts limit
to 1, it's a single read by column. The other query with limit 10 is
translated internally to a slice query and may explain the CPU difference

 Now, do not take my words as granted. Those points are just *assumptions *and
partial conclusions. I need extensive in depth debugging to confirm those.
Did not have time lately.

On Tue, Sep 23, 2014 at 10:46 PM, Chris Lohfink 
wrote:

> CPU consumption may be affected from the cassandra-stress tool in 2nd
> example as well.  Running on a separate system eliminates it as a possible
> cause.  There is a little extra work but not anything that I think would be
> that obvious.  tracing (can enable with nodetool) or profiling (ie with
> yourkit) can give more exposure to the bottleneck.  Id run test from
> separate system first.
>
> ---
> Chris Lohfink
>
>
> On Sep 23, 2014, at 12:48 PM, Leleu Eric  wrote:
>
> First of all, Thanks for your help ! :)
>
> Here is some details :
>
> With RF=N=2 your essentially testing a single machine locally which isnt
> the best indicator long term
>
> I will  test with more nodes, (4 with RF = 2) but for now I'm limited to 2
> nodes for non technical reason ...
>
> Well, first off you shouldn't run stress tool on the node your testing.
> Give it its own box.
>
> I performed the test in a new Keyspace in order to have a clear dataset.
>
> the 2nd query since its returning 10x the data and there will be more to
> go through within the partition
>
> I configured cassandra-stress in a way of each user has only one bucket so
> the amount of data is the same in the both case. ("select * from buckets
> where name = ? and tenantid = ? limit 1" and "select * from
> owner_to_buckets  where owner = ? and tenantid = ? limit 10").
> Does cassandra perform extra read when the limit is bigger than the
> available data (even if the partition key contains only one single value in
> the clustering column) ?
> If the amount of data is the same, how can we explain the difference of
> CPU consumption?
>
>
> Regards,
> Eric
>
> 
> De : Chris Lohfink [clohf...@blackbirdit.com]
> Date d'envoi : mardi 23 septembre 2014 19:23
> À : user@cassandra.apache.org
> Objet : Re: CPU consumption of Cassandra
>
> Well, first off you shouldn't run stress tool on the node your testing.
> Give it its own box.
>
> With RF=N=2 your essentially testing a single machine locally which isnt
> the best indicator long term (optimizations available when reading data
> thats local to the node).  80k/sec on a system is pretty good though, your
> probably seeing slower on the 2nd query since its returning 10x the data
> and there will be more to go through within the partition. 42k/sec is still
> acceptable imho since these are smaller boxes.  You are probably seeing
> high CPU because the system is doing a lot :)
>
> If you want to get more out of these systems can do some tuning probably,
> enable trace to see whats actually the bottleneck.
>
> Collections will very likely hurt more then help.
>
> ---
> Chris Lohfink
>
> On Sep 23, 2014, at 9:39 AM, Leleu Eric  mailto:eric.le...@worldline.com >> wrote:
>
> I tried to run “cassandra-stress” on some of my table as proposed by Jake
> Luciani.
>
> For a simple table, this tool is able to perform 8 read op/s with a
> few CPU consumption if I request the table by the PK(name, tenanted)
>
> Ex :
> TABLE :
>
> CREATE TABLE IF NOT EXISTS buckets (tenantid varchar,
> name varchar,
> owner varchar,
> location varchar,
> description varchar,
> codeQuota varchar,
> creationDate timestamp,
> updateDate timestamp,
> PRIMARY KEY (name, tenantid));
>
> QUERY : select * from buckets where name = ? and tenantid = ? limit 1;
>
> TOP output for 900 threads on cassandra-stress :
> top - 13:17:09 up 173 days, 21:54,  4 users,  load average: 11.88, 4.30,
> 2.76
> Tasks: 272 total,   1 running, 270 sleeping,   0 stopped,   1 zombie
> Cpu(s): 71.4%us, 14.0%sy,  0.0%ni, 13.1%id,  0.0%wa,  0.0%hi,  1.5%si,
>  0.0%st
> Mem:  98894704k total, 96367436k used,  2527268k free,15440k buffers
> Swap:0k total,0k used,0k free, 88194556k cached
>
>  PID USER  PR  NI  VIRT  RES  SHR S %CPU %MEMTIME+  COMMAND
> 25857 root  20   0 29.7g 1.5g  12m S 693.0  1.6  38:45.58 java  <==
> Cassandra-stress
> 29160 cass

Re: CPU consumption of Cassandra

[Chris Lohfink]

CPU consumption may be affected from the cassandra-stress tool in 2nd example 
as well.  Running on a separate system eliminates it as a possible cause.  
There is a little extra work but not anything that I think would be that 
obvious.  tracing (can enable with nodetool) or profiling (ie with yourkit) can 
give more exposure to the bottleneck.  Id run test from separate system first.

---
Chris Lohfink 


On Sep 23, 2014, at 12:48 PM, Leleu Eric  wrote:

> First of all, Thanks for your help ! :)
> 
> Here is some details :
> 
>> With RF=N=2 your essentially testing a single machine locally which isnt the 
>> best indicator long term
> I will  test with more nodes, (4 with RF = 2) but for now I'm limited to 2 
> nodes for non technical reason ...
> 
>> Well, first off you shouldn't run stress tool on the node your testing.  
>> Give it its own box.
> I performed the test in a new Keyspace in order to have a clear dataset.
> 
>> the 2nd query since its returning 10x the data and there will be more to go 
>> through within the partition
> I configured cassandra-stress in a way of each user has only one bucket so 
> the amount of data is the same in the both case. ("select * from buckets 
> where name = ? and tenantid = ? limit 1" and "select * from owner_to_buckets  
> where owner = ? and tenantid = ? limit 10").
> Does cassandra perform extra read when the limit is bigger than the available 
> data (even if the partition key contains only one single value in the 
> clustering column) ?
> If the amount of data is the same, how can we explain the difference of CPU 
> consumption?
> 
> 
> Regards,
> Eric
> 
> 
> De : Chris Lohfink [clohf...@blackbirdit.com]
> Date d'envoi : mardi 23 septembre 2014 19:23
> À : user@cassandra.apache.org
> Objet : Re: CPU consumption of Cassandra
> 
> Well, first off you shouldn't run stress tool on the node your testing.  Give 
> it its own box.
> 
> With RF=N=2 your essentially testing a single machine locally which isnt the 
> best indicator long term (optimizations available when reading data thats 
> local to the node).  80k/sec on a system is pretty good though, your probably 
> seeing slower on the 2nd query since its returning 10x the data and there 
> will be more to go through within the partition. 42k/sec is still acceptable 
> imho since these are smaller boxes.  You are probably seeing high CPU because 
> the system is doing a lot :)
> 
> If you want to get more out of these systems can do some tuning probably, 
> enable trace to see whats actually the bottleneck.
> 
> Collections will very likely hurt more then help.
> 
> ---
> Chris Lohfink
> 
> On Sep 23, 2014, at 9:39 AM, Leleu Eric 
> mailto:eric.le...@worldline.com>> wrote:
> 
> I tried to run “cassandra-stress” on some of my table as proposed by Jake 
> Luciani.
> 
> For a simple table, this tool is able to perform 8 read op/s with a few 
> CPU consumption if I request the table by the PK(name, tenanted)
> 
> Ex :
> TABLE :
> 
> CREATE TABLE IF NOT EXISTS buckets (tenantid varchar,
> name varchar,
> owner varchar,
> location varchar,
> description varchar,
> codeQuota varchar,
> creationDate timestamp,
> updateDate timestamp,
> PRIMARY KEY (name, tenantid));
> 
> QUERY : select * from buckets where name = ? and tenantid = ? limit 1;
> 
> TOP output for 900 threads on cassandra-stress :
> top - 13:17:09 up 173 days, 21:54,  4 users,  load average: 11.88, 4.30, 2.76
> Tasks: 272 total,   1 running, 270 sleeping,   0 stopped,   1 zombie
> Cpu(s): 71.4%us, 14.0%sy,  0.0%ni, 13.1%id,  0.0%wa,  0.0%hi,  1.5%si,  0.0%st
> Mem:  98894704k total, 96367436k used,  2527268k free,15440k buffers
> Swap:0k total,0k used,0k free, 88194556k cached
> 
>  PID USER  PR  NI  VIRT  RES  SHR S %CPU %MEMTIME+  COMMAND
> 25857 root  20   0 29.7g 1.5g  12m S 693.0  1.6  38:45.58 java  <== 
> Cassandra-stress
> 29160 cassandr  20   0 16.3g 4.8g  10m S  1.3  5.0  44:46.89 java  <== 
> Cassandra
> 
> 
> 
> Now, If I run another query on a table that provides a list of buckets 
> according to the  owner, the number of op/s is divided by 2  (42000 op/s) and 
> CPU consumption grow UP.
> 
> Ex :
> TABLE :
> 
> CREATE TABLE IF NOT EXISTS owner_to_buckets (tenantid varchar,
> name varchar,
> owner varchar,
> location varchar,
> description varchar,
> codeQuota varchar,
> creationDate timestamp,
> updateDate timestamp,
> PRIMARY KEY ((owner, tenantid), name));
> 
> QUERY : select * from owner_to_buckets  where owner = ? and tenantid = ? 
> limit 10;
> 
> TOP output for 4  threads on cassandra-stress:
> 
> top - 13:49:16 up 173 days, 22:26,  4 users,  load average: 1.76, 1.48, 1.17
> Tasks: 273 total,   1 running, 271 sleeping,   0 stopped,   1 zombie
> Cpu(s): 26.3%us,  8.0%sy,  0.0%ni, 64.7%id,  0.0%wa,  0.0%hi,  1.0%si,  0.0%st
> Mem:  98894704k total, 97512156k used,  1382548k free,14580k buffers
> Swap:0k total,0k used,0k free, 

Re: Is there harm from having all the nodes in the seed list?

[DuyHai Doan]

Well, having all nodes in the seed list does not compromise any correctness
of gossip protocol. However there will be extra network traffic when nodes
are starting because it will ping all nodes for topology discovery, AFAIK

On Tue, Sep 23, 2014 at 7:31 PM, Donald Smith <
donald.sm...@audiencescience.com> wrote:

>  Is there any harm from having all the nodes listed in the seeds list in
> cassandra.yaml?
>
>
>
> *Donald A. Smith* | Senior Software Engineer
> P: 425.201.3900 x 3866
> C: (206) 819-5965
> F: (646) 443-2333
> dona...@audiencescience.com
>
>
> [image: AudienceScience]
>
>
>

RE : CPU consumption of Cassandra

[Leleu Eric]

First of all, Thanks for your help ! :)

Here is some details :

> With RF=N=2 your essentially testing a single machine locally which isnt the 
> best indicator long term
I will  test with more nodes, (4 with RF = 2) but for now I'm limited to 2 
nodes for non technical reason ...

> Well, first off you shouldn't run stress tool on the node your testing.  Give 
> it its own box.
I performed the test in a new Keyspace in order to have a clear dataset.

>  the 2nd query since its returning 10x the data and there will be more to go 
> through within the partition
I configured cassandra-stress in a way of each user has only one bucket so the 
amount of data is the same in the both case. ("select * from buckets where name 
= ? and tenantid = ? limit 1" and "select * from owner_to_buckets  where owner 
= ? and tenantid = ? limit 10").
Does cassandra perform extra read when the limit is bigger than the available 
data (even if the partition key contains only one single value in the 
clustering column) ?
If the amount of data is the same, how can we explain the difference of CPU 
consumption?


Regards,
Eric


De : Chris Lohfink [clohf...@blackbirdit.com]
Date d'envoi : mardi 23 septembre 2014 19:23
À : user@cassandra.apache.org
Objet : Re: CPU consumption of Cassandra

Well, first off you shouldn't run stress tool on the node your testing.  Give 
it its own box.

With RF=N=2 your essentially testing a single machine locally which isnt the 
best indicator long term (optimizations available when reading data thats local 
to the node).  80k/sec on a system is pretty good though, your probably seeing 
slower on the 2nd query since its returning 10x the data and there will be more 
to go through within the partition. 42k/sec is still acceptable imho since 
these are smaller boxes.  You are probably seeing high CPU because the system 
is doing a lot :)

If you want to get more out of these systems can do some tuning probably, 
enable trace to see whats actually the bottleneck.

Collections will very likely hurt more then help.

---
Chris Lohfink

On Sep 23, 2014, at 9:39 AM, Leleu Eric 
mailto:eric.le...@worldline.com>> wrote:

I tried to run “cassandra-stress” on some of my table as proposed by Jake 
Luciani.

For a simple table, this tool is able to perform 8 read op/s with a few CPU 
consumption if I request the table by the PK(name, tenanted)

Ex :
TABLE :

CREATE TABLE IF NOT EXISTS buckets (tenantid varchar,
name varchar,
owner varchar,
location varchar,
description varchar,
codeQuota varchar,
creationDate timestamp,
updateDate timestamp,
PRIMARY KEY (name, tenantid));

QUERY : select * from buckets where name = ? and tenantid = ? limit 1;

TOP output for 900 threads on cassandra-stress :
top - 13:17:09 up 173 days, 21:54,  4 users,  load average: 11.88, 4.30, 2.76
Tasks: 272 total,   1 running, 270 sleeping,   0 stopped,   1 zombie
Cpu(s): 71.4%us, 14.0%sy,  0.0%ni, 13.1%id,  0.0%wa,  0.0%hi,  1.5%si,  0.0%st
Mem:  98894704k total, 96367436k used,  2527268k free,15440k buffers
Swap:0k total,0k used,0k free, 88194556k cached

  PID USER  PR  NI  VIRT  RES  SHR S %CPU %MEMTIME+  COMMAND
25857 root  20   0 29.7g 1.5g  12m S 693.0  1.6  38:45.58 java  <== 
Cassandra-stress
29160 cassandr  20   0 16.3g 4.8g  10m S  1.3  5.0  44:46.89 java  <== Cassandra



Now, If I run another query on a table that provides a list of buckets 
according to the  owner, the number of op/s is divided by 2  (42000 op/s) and 
CPU consumption grow UP.

Ex :
TABLE :

CREATE TABLE IF NOT EXISTS owner_to_buckets (tenantid varchar,
name varchar,
owner varchar,
location varchar,
description varchar,
codeQuota varchar,
creationDate timestamp,
updateDate timestamp,
PRIMARY KEY ((owner, tenantid), name));

QUERY : select * from owner_to_buckets  where owner = ? and tenantid = ? limit 
10;

TOP output for 4  threads on cassandra-stress:

top - 13:49:16 up 173 days, 22:26,  4 users,  load average: 1.76, 1.48, 1.17
Tasks: 273 total,   1 running, 271 sleeping,   0 stopped,   1 zombie
Cpu(s): 26.3%us,  8.0%sy,  0.0%ni, 64.7%id,  0.0%wa,  0.0%hi,  1.0%si,  0.0%st
Mem:  98894704k total, 97512156k used,  1382548k free,14580k buffers
Swap:0k total,0k used,0k free, 90413772k cached

  PID USER  PR  NI  VIRT  RES  SHR S %CPU %MEMTIME+  COMMAND
29160 cassandr  20   0 13.6g 4.8g  37m S 186.7  5.1  62:26.77 java <== Cassandra
50622 root  20   0 28.8g 469m  12m S 102.5  0.5   0:45.84 java <== 
Cassandra-stress

TOP output for 271  threads on cassandra-stress:


top - 13:57:03 up 173 days, 22:34,  4 users,  load average: 4.67, 1.76, 1.25
Tasks: 272 total,   1 running, 270 sleeping,   0 stopped,   1 zombie
Cpu(s): 81.5%us, 14.0%sy,  0.0%ni,  3.1%id,  0.0%wa,  0.0%hi,  1.3%si,  0.0%st
Mem:  98894704k total, 94955936k used,  3938768k free,15892k buffers
Swap:0k total,0k used,0k free, 85993676k cached

  PID USER

Cassandra sometimes times out on write queries and it spends majority amount of the CPU time on method org.apache.cassandra.db.marshal.AbstractCompositeType.compare()

[Li, George]

Hi,

I am running some load test in a 5 node Cassandra cluster (EC2, single
region, each node has 15 GB RAM, Cassandra version 2.0.6, replication
factor 3). My Java program uses Java driver version 2.0.6 and it does 2000
rounds of batch write queries, each with 8 inserts, 8 updates and 8
deletes. When I run the test, I usually see really high CPU usage on some
of the nodes. And sometimes I get a time out indicated by
com.datastax.driver.core.exceptions.WriteTimeoutException: Cassandra
timeout during write query at consistency ONE (1 replica were required but
only 0 acknowledged the write). My cluster has 1 ms write timeout limit.
I used VisualVM and did some CPU sampling. What I noticed is that CPU
spends majority of the time inside method
org.apache.cassandra.db.marshal.AbstractCompositeType.compare()
and the snapshot indicates that the call tree is most of the time like this:

java.lang.Thread.State: RUNNABLE
at
org.apache.cassandra.db.marshal.AbstractCompositeType.compare(AbstractCompositeType.java:98)
at
org.apache.cassandra.db.marshal.AbstractCompositeType.compare(AbstractCompositeType.java:35)
at java.util.Arrays.binarySearch0(Arrays.java:1585)
at java.util.Arrays.binarySearch(Arrays.java:1570)
at
org.apache.cassandra.db.RangeTombstoneList.searchInternal(RangeTombstoneList.java:236)
at
org.apache.cassandra.db.RangeTombstoneList.isDeleted(RangeTombstoneList.java:210)
at org.apache.cassandra.db.DeletionInfo.isDeleted(DeletionInfo.java:136)
at org.apache.cassandra.db.DeletionInfo.isDeleted(DeletionInfo.java:123)
at
org.apache.cassandra.db.AtomicSortedColumns.addAllWithSizeDelta(AtomicSortedColumns.java:193)
at org.apache.cassandra.db.Memtable.resolve(Memtable.java:194)
at org.apache.cassandra.db.Memtable.put(Memtable.java:158)
at
org.apache.cassandra.db.ColumnFamilyStore.apply(ColumnFamilyStore.java:891)
at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:368)
at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:333)
at org.apache.cassandra.db.RowMutation.apply(RowMutation.java:206)
at
org.apache.cassandra.db.RowMutationVerbHandler.doVerb(RowMutationVerbHandler.java:56)
at
org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:60)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)

So it looks like it is tombstone search related even though I am not seeing
tombstone warning message in Cassandra's system log.
Per
http://www.datastax.com/dev/blog/how-cassandra-deals-with-replica-failure,
write timeout is not really an error. But with my load and 10 seconds write
timeout limit, getting timeout seems to be problematic. Do you guys have
some hint or pointer what I should go about this issue?

Thanks.

Is there harm from having all the nodes in the seed list?

[Donald Smith]

Is there any harm from having all the nodes listed in the seeds list in 
cassandra.yaml?

Donald A. Smith | Senior Software Engineer
P: 425.201.3900 x 3866
C: (206) 819-5965
F: (646) 443-2333
dona...@audiencescience.com

[AudienceScience]

Re: CPU consumption of Cassandra

[Chris Lohfink]

Well, first off you shouldn't run stress tool on the node your testing.  Give 
it its own box.  

With RF=N=2 your essentially testing a single machine locally which isnt the 
best indicator long term (optimizations available when reading data thats local 
to the node).  80k/sec on a system is pretty good though, your probably seeing 
slower on the 2nd query since its returning 10x the data and there will be more 
to go through within the partition. 42k/sec is still acceptable imho since 
these are smaller boxes.  You are probably seeing high CPU because the system 
is doing a lot :)

If you want to get more out of these systems can do some tuning probably, 
enable trace to see whats actually the bottleneck. 

Collections will very likely hurt more then help.

---
Chris Lohfink

On Sep 23, 2014, at 9:39 AM, Leleu Eric  wrote:

> I tried to run “cassandra-stress” on some of my table as proposed by Jake 
> Luciani.
>  
> For a simple table, this tool is able to perform 8 read op/s with a few 
> CPU consumption if I request the table by the PK(name, tenanted)
>  
> Ex :  
> TABLE :
>  
> CREATE TABLE IF NOT EXISTS buckets (tenantid varchar,
> name varchar,
> owner varchar,
> location varchar,
> description varchar,
> codeQuota varchar,
> creationDate timestamp,
> updateDate timestamp,
> PRIMARY KEY (name, tenantid));
>  
> QUERY : select * from buckets where name = ? and tenantid = ? limit 1;
>  
> TOP output for 900 threads on cassandra-stress :
> top - 13:17:09 up 173 days, 21:54,  4 users,  load average: 11.88, 4.30, 2.76
> Tasks: 272 total,   1 running, 270 sleeping,   0 stopped,   1 zombie
> Cpu(s): 71.4%us, 14.0%sy,  0.0%ni, 13.1%id,  0.0%wa,  0.0%hi,  1.5%si,  0.0%st
> Mem:  98894704k total, 96367436k used,  2527268k free,15440k buffers
> Swap:0k total,0k used,0k free, 88194556k cached
>  
>   PID USER  PR  NI  VIRT  RES  SHR S %CPU %MEMTIME+  COMMAND
> 25857 root  20   0 29.7g 1.5g  12m S 693.0  1.6  38:45.58 java  ç 
> Cassandra-stress
> 29160 cassandr  20   0 16.3g 4.8g  10m S  1.3  5.0  44:46.89 java  ç Cassandra
>  
>  
>  
> Now, If I run another query on a table that provides a list of buckets 
> according to the  owner, the number of op/s is divided by 2  (42000 op/s) and 
> CPU consumption grow UP.
>  
> Ex :  
> TABLE :
>  
> CREATE TABLE IF NOT EXISTS owner_to_buckets (tenantid varchar,
> name varchar,
> owner varchar,
> location varchar,
> description varchar,
> codeQuota varchar,
> creationDate timestamp,
> updateDate timestamp,
> PRIMARY KEY ((owner, tenantid), name));
>  
> QUERY : select * from owner_to_buckets  where owner = ? and tenantid = ? 
> limit 10;
>  
> TOP output for 4  threads on cassandra-stress:
>  
> top - 13:49:16 up 173 days, 22:26,  4 users,  load average: 1.76, 1.48, 1.17
> Tasks: 273 total,   1 running, 271 sleeping,   0 stopped,   1 zombie
> Cpu(s): 26.3%us,  8.0%sy,  0.0%ni, 64.7%id,  0.0%wa,  0.0%hi,  1.0%si,  0.0%st
> Mem:  98894704k total, 97512156k used,  1382548k free,14580k buffers
> Swap:0k total,0k used,0k free, 90413772k cached
>  
>   PID USER  PR  NI  VIRT  RES  SHR S %CPU %MEMTIME+  COMMAND
> 29160 cassandr  20   0 13.6g 4.8g  37m S 186.7  5.1  62:26.77 java ç Cassandra
> 50622 root  20   0 28.8g 469m  12m S 102.5  0.5   0:45.84 java ç 
> Cassandra-stress
>  
> TOP output for 271  threads on cassandra-stress:
>  
>  
> top - 13:57:03 up 173 days, 22:34,  4 users,  load average: 4.67, 1.76, 1.25
> Tasks: 272 total,   1 running, 270 sleeping,   0 stopped,   1 zombie
> Cpu(s): 81.5%us, 14.0%sy,  0.0%ni,  3.1%id,  0.0%wa,  0.0%hi,  1.3%si,  0.0%st
> Mem:  98894704k total, 94955936k used,  3938768k free,15892k buffers
> Swap:0k total,0k used,0k free, 85993676k cached
>  
>   PID USER  PR  NI  VIRT  RES  SHR S %CPU %MEMTIME+  COMMAND
> 29160 cassandr  20   0 13.6g 4.8g  38m S 430.0  5.1  82:31.80 java ç Cassandra
> 50622 root  20   0 29.1g 2.3g  12m S 343.4  2.4  17:51.22 java ç 
> Cassandra-stress
>  
>  
> I have 4 tables with  a composed PRIMARY KEY (two of them has 4 entries : 2 
> for the partition key, one for cluster column and one for sort column)
> Two of these tables are frequently read with the partition key because we 
> want to list data of a given user, this should explain my CPU load according 
> to the simple test done with Cassandra-stress …
>  
> How can I avoid this?
> Collections could be an option but the number of data per user is not limited 
> and can easily exceed 200 entries. According to the Cassandra documentation, 
> collections have a size limited to 64KB. So it is probably not a solution in 
> my case. L
>  
>  
> Regards,
> Eric
>  
> De : Chris Lohfink [mailto:clohf...@blackbirdit.com] 
> Envoyé : lundi 22 septembre 2014 22:03
> À : user@cassandra.apache.org
> Objet : Re: CPU consumption of Cassandra
>  
> Its going to depend a lot on your data model but 5-6k is on the low end of 
> what I would expect.  

RE: CPU consumption of Cassandra

[Leleu Eric]

I tried to run "cassandra-stress" on some of my table as proposed by Jake 
Luciani.

For a simple table, this tool is able to perform 8 read op/s with a few CPU 
consumption if I request the table by the PK(name, tenanted)

Ex :
TABLE :

CREATE TABLE IF NOT EXISTS buckets (tenantid varchar,
name varchar,
owner varchar,
location varchar,
description varchar,
codeQuota varchar,
creationDate timestamp,
updateDate timestamp,
PRIMARY KEY (name, tenantid));

QUERY : select * from buckets where name = ? and tenantid = ? limit 1;

TOP output for 900 threads on cassandra-stress :
top - 13:17:09 up 173 days, 21:54,  4 users,  load average: 11.88, 4.30, 2.76
Tasks: 272 total,   1 running, 270 sleeping,   0 stopped,   1 zombie
Cpu(s): 71.4%us, 14.0%sy,  0.0%ni, 13.1%id,  0.0%wa,  0.0%hi,  1.5%si,  0.0%st
Mem:  98894704k total, 96367436k used,  2527268k free,15440k buffers
Swap:0k total,0k used,0k free, 88194556k cached

  PID USER  PR  NI  VIRT  RES  SHR S %CPU %MEMTIME+  COMMAND
25857 root  20   0 29.7g 1.5g  12m S 693.0  1.6  38:45.58 java  <== 
Cassandra-stress
29160 cassandr  20   0 16.3g 4.8g  10m S  1.3  5.0  44:46.89 java  <== Cassandra



Now, If I run another query on a table that provides a list of buckets 
according to the  owner, the number of op/s is divided by 2  (42000 op/s) and 
CPU consumption grow UP.

Ex :
TABLE :

CREATE TABLE IF NOT EXISTS owner_to_buckets (tenantid varchar,
name varchar,
owner varchar,
location varchar,
description varchar,
codeQuota varchar,
creationDate timestamp,
updateDate timestamp,
PRIMARY KEY ((owner, tenantid), name));

QUERY : select * from owner_to_buckets  where owner = ? and tenantid = ? limit 
10;

TOP output for 4  threads on cassandra-stress:

top - 13:49:16 up 173 days, 22:26,  4 users,  load average: 1.76, 1.48, 1.17
Tasks: 273 total,   1 running, 271 sleeping,   0 stopped,   1 zombie
Cpu(s): 26.3%us,  8.0%sy,  0.0%ni, 64.7%id,  0.0%wa,  0.0%hi,  1.0%si,  0.0%st
Mem:  98894704k total, 97512156k used,  1382548k free,14580k buffers
Swap:0k total,0k used,0k free, 90413772k cached

  PID USER  PR  NI  VIRT  RES  SHR S %CPU %MEMTIME+  COMMAND
29160 cassandr  20   0 13.6g 4.8g  37m S 186.7  5.1  62:26.77 java <== Cassandra
50622 root  20   0 28.8g 469m  12m S 102.5  0.5   0:45.84 java <== 
Cassandra-stress

TOP output for 271  threads on cassandra-stress:


top - 13:57:03 up 173 days, 22:34,  4 users,  load average: 4.67, 1.76, 1.25
Tasks: 272 total,   1 running, 270 sleeping,   0 stopped,   1 zombie
Cpu(s): 81.5%us, 14.0%sy,  0.0%ni,  3.1%id,  0.0%wa,  0.0%hi,  1.3%si,  0.0%st
Mem:  98894704k total, 94955936k used,  3938768k free,15892k buffers
Swap:0k total,0k used,0k free, 85993676k cached

  PID USER  PR  NI  VIRT  RES  SHR S %CPU %MEMTIME+  COMMAND
29160 cassandr  20   0 13.6g 4.8g  38m S 430.0  5.1  82:31.80 java <== Cassandra
50622 root  20   0 29.1g 2.3g  12m S 343.4  2.4  17:51.22 java <== 
Cassandra-stress


I have 4 tables with  a composed PRIMARY KEY (two of them has 4 entries : 2 for 
the partition key, one for cluster column and one for sort column)
Two of these tables are frequently read with the partition key because we want 
to list data of a given user, this should explain my CPU load according to the 
simple test done with Cassandra-stress ...

How can I avoid this?
Collections could be an option but the number of data per user is not limited 
and can easily exceed 200 entries. According to the Cassandra documentation, 
collections have a size limited to 64KB. So it is probably not a solution in my 
case. :(


Regards,
Eric

De : Chris Lohfink [mailto:clohf...@blackbirdit.com]
Envoyé : lundi 22 septembre 2014 22:03
À : user@cassandra.apache.org
Objet : Re: CPU consumption of Cassandra

Its going to depend a lot on your data model but 5-6k is on the low end of what 
I would expect.  N=RF=2 is not really something I would recommend.  That said 
93GB is not much data so the bottleneck may exist more in your data model, 
queries, or client.

What profiler are you using?  The cpu on the select/read is marked as RUNNABLE 
but its really more of a wait state that may throw some profilers off, it may 
be a red haring.

---
Chris Lohfink

On Sep 22, 2014, at 11:39 AM, Leleu Eric 
mailto:eric.le...@worldline.com>> wrote:


Hi,


I'm currently testing Cassandra 2.0.9  (and since the last week 2.1) under some 
read heavy load...

I have 2 cassandra nodes (RF : 2) running under CentOS 6 with 16GB of RAM and 8 
Cores.
I have around 93GB of data per node (one Disk of 300GB with SAS interface and a 
Rotational Speed of 10500)

I have 300 active client threads and they request the C* nodes with a 
Consitency level set to ONE (I'm using the CQL datastax driver).

During my tests I saw  a lot of CPU consumption (70% user / 6%sys / 4% iowait / 
20%idle).
C* nodes respond to around 5000 op/s (sometime up to 6000op/s)

I try to profile a node and at