CPU consumption may be affected from the cassandra-stress tool in 2nd example 
as well.  Running on a separate system eliminates it as a possible cause.  
There is a little extra work but not anything that I think would be that 
obvious.  tracing (can enable with nodetool) or profiling (ie with yourkit) can 
give more exposure to the bottleneck.  Id run test from separate system first.

---
Chris Lohfink 


On Sep 23, 2014, at 12:48 PM, Leleu Eric <eric.le...@worldline.com> wrote:

> First of all, Thanks for your help ! :)
> 
> Here is some details :
> 
>> With RF=N=2 your essentially testing a single machine locally which isnt the 
>> best indicator long term
> I will  test with more nodes, (4 with RF = 2) but for now I'm limited to 2 
> nodes for non technical reason ...
> 
>> Well, first off you shouldn't run stress tool on the node your testing.  
>> Give it its own box.
> I performed the test in a new Keyspace in order to have a clear dataset.
> 
>> the 2nd query since its returning 10x the data and there will be more to go 
>> through within the partition
> I configured cassandra-stress in a way of each user has only one bucket so 
> the amount of data is the same in the both case. ("select * from buckets 
> where name = ? and tenantid = ? limit 1" and "select * from owner_to_buckets  
> where owner = ? and tenantid = ? limit 10").
> Does cassandra perform extra read when the limit is bigger than the available 
> data (even if the partition key contains only one single value in the 
> clustering column) ?
> If the amount of data is the same, how can we explain the difference of CPU 
> consumption?
> 
> 
> Regards,
> Eric
> 
> ________________________________________
> De : Chris Lohfink [clohf...@blackbirdit.com]
> Date d'envoi : mardi 23 septembre 2014 19:23
> À : user@cassandra.apache.org
> Objet : Re: CPU consumption of Cassandra
> 
> Well, first off you shouldn't run stress tool on the node your testing.  Give 
> it its own box.
> 
> With RF=N=2 your essentially testing a single machine locally which isnt the 
> best indicator long term (optimizations available when reading data thats 
> local to the node).  80k/sec on a system is pretty good though, your probably 
> seeing slower on the 2nd query since its returning 10x the data and there 
> will be more to go through within the partition. 42k/sec is still acceptable 
> imho since these are smaller boxes.  You are probably seeing high CPU because 
> the system is doing a lot :)
> 
> If you want to get more out of these systems can do some tuning probably, 
> enable trace to see whats actually the bottleneck.
> 
> Collections will very likely hurt more then help.
> 
> ---
> Chris Lohfink
> 
> On Sep 23, 2014, at 9:39 AM, Leleu Eric 
> <eric.le...@worldline.com<mailto:eric.le...@worldline.com>> wrote:
> 
> I tried to run “cassandra-stress” on some of my table as proposed by Jake 
> Luciani.
> 
> For a simple table, this tool is able to perform 80000 read op/s with a few 
> CPU consumption if I request the table by the PK(name, tenanted)
> 
> Ex :
> TABLE :
> 
> CREATE TABLE IF NOT EXISTS buckets (tenantid varchar,
> name varchar,
> owner varchar,
> location varchar,
> description varchar,
> codeQuota varchar,
> creationDate timestamp,
> updateDate timestamp,
> PRIMARY KEY (name, tenantid));
> 
> QUERY : select * from buckets where name = ? and tenantid = ? limit 1;
> 
> TOP output for 900 threads on cassandra-stress :
> top - 13:17:09 up 173 days, 21:54,  4 users,  load average: 11.88, 4.30, 2.76
> Tasks: 272 total,   1 running, 270 sleeping,   0 stopped,   1 zombie
> Cpu(s): 71.4%us, 14.0%sy,  0.0%ni, 13.1%id,  0.0%wa,  0.0%hi,  1.5%si,  0.0%st
> Mem:  98894704k total, 96367436k used,  2527268k free,    15440k buffers
> Swap:        0k total,        0k used,        0k free, 88194556k cached
> 
>  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
> 25857 root      20   0 29.7g 1.5g  12m S 693.0  1.6  38:45.58 java  <== 
> Cassandra-stress
> 29160 cassandr  20   0 16.3g 4.8g  10m S  1.3  5.0  44:46.89 java  <== 
> Cassandra
> 
> 
> 
> Now, If I run another query on a table that provides a list of buckets 
> according to the  owner, the number of op/s is divided by 2  (42000 op/s) and 
> CPU consumption grow UP.
> 
> Ex :
> TABLE :
> 
> CREATE TABLE IF NOT EXISTS owner_to_buckets (tenantid varchar,
> name varchar,
> owner varchar,
> location varchar,
> description varchar,
> codeQuota varchar,
> creationDate timestamp,
> updateDate timestamp,
> PRIMARY KEY ((owner, tenantid), name));
> 
> QUERY : select * from owner_to_buckets  where owner = ? and tenantid = ? 
> limit 10;
> 
> TOP output for 4  threads on cassandra-stress:
> 
> top - 13:49:16 up 173 days, 22:26,  4 users,  load average: 1.76, 1.48, 1.17
> Tasks: 273 total,   1 running, 271 sleeping,   0 stopped,   1 zombie
> Cpu(s): 26.3%us,  8.0%sy,  0.0%ni, 64.7%id,  0.0%wa,  0.0%hi,  1.0%si,  0.0%st
> Mem:  98894704k total, 97512156k used,  1382548k free,    14580k buffers
> Swap:        0k total,        0k used,        0k free, 90413772k cached
> 
>  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
> 29160 cassandr  20   0 13.6g 4.8g  37m S 186.7  5.1  62:26.77 java <== 
> Cassandra
> 50622 root      20   0 28.8g 469m  12m S 102.5  0.5   0:45.84 java <== 
> Cassandra-stress
> 
> TOP output for 271  threads on cassandra-stress:
> 
> 
> top - 13:57:03 up 173 days, 22:34,  4 users,  load average: 4.67, 1.76, 1.25
> Tasks: 272 total,   1 running, 270 sleeping,   0 stopped,   1 zombie
> Cpu(s): 81.5%us, 14.0%sy,  0.0%ni,  3.1%id,  0.0%wa,  0.0%hi,  1.3%si,  0.0%st
> Mem:  98894704k total, 94955936k used,  3938768k free,    15892k buffers
> Swap:        0k total,        0k used,        0k free, 85993676k cached
> 
>  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
> 29160 cassandr  20   0 13.6g 4.8g  38m S 430.0  5.1  82:31.80 java <== 
> Cassandra
> 50622 root      20   0 29.1g 2.3g  12m S 343.4  2.4  17:51.22 java <== 
> Cassandra-stress
> 
> 
> I have 4 tables with  a composed PRIMARY KEY (two of them has 4 entries : 2 
> for the partition key, one for cluster column and one for sort column)
> Two of these tables are frequently read with the partition key because we 
> want to list data of a given user, this should explain my CPU load according 
> to the simple test done with Cassandra-stress …
> 
> How can I avoid this?
> Collections could be an option but the number of data per user is not limited 
> and can easily exceed 200 entries. According to the Cassandra documentation, 
> collections have a size limited to 64KB. So it is probably not a solution in 
> my case. ☹
> 
> 
> Regards,
> Eric
> 
> De : Chris Lohfink [mailto:clohf...@blackbirdit.com]
> Envoyé : lundi 22 septembre 2014 22:03
> À : user@cassandra.apache.org<mailto:user@cassandra.apache.org>
> Objet : Re: CPU consumption of Cassandra
> 
> Its going to depend a lot on your data model but 5-6k is on the low end of 
> what I would expect.  N=RF=2 is not really something I would recommend.  That 
> said 93GB is not much data so the bottleneck may exist more in your data 
> model, queries, or client.
> 
> What profiler are you using?  The cpu on the select/read is marked as 
> RUNNABLE but its really more of a wait state that may throw some profilers 
> off, it may be a red haring.
> 
> ---
> Chris Lohfink
> 
> On Sep 22, 2014, at 11:39 AM, Leleu Eric 
> <eric.le...@worldline.com<mailto:eric.le...@worldline.com>> wrote:
> 
> 
> Hi,
> 
> 
> I’m currently testing Cassandra 2.0.9  (and since the last week 2.1) under 
> some read heavy load…
> 
> I have 2 cassandra nodes (RF : 2) running under CentOS 6 with 16GB of RAM and 
> 8 Cores.
> I have around 93GB of data per node (one Disk of 300GB with SAS interface and 
> a Rotational Speed of 10500)
> 
> I have 300 active client threads and they request the C* nodes with a 
> Consitency level set to ONE (I’m using the CQL datastax driver).
> 
> During my tests I saw  a lot of CPU consumption (70% user / 6%sys / 4% iowait 
> / 20%idle).
> C* nodes respond to around 5000 op/s (sometime up to 6000op/s)
> 
> I try to profile a node and at the first look, 60% of the CPU is passed in 
> the “sun.nio.ch<http://sun.nio.ch/>” package. (SelectorImpl.select or 
> Channel.read)
> 
> I know that Benchmark results are highly dependent of the Dataset and use 
> cases, but according to my point of view this CPU consumption is normal 
> according to the load.
> Someone can confirm that point ?
> According to my Hardware configuration, can I expect to have more than 6000 
> read op/s ?
> 
> 
> Regards,
> Eric
> 
> 
> 
> 
> 
> ________________________________
> 
> Ce message et les pièces jointes sont confidentiels et réservés à l'usage 
> exclusif de ses destinataires. Il peut également être protégé par le secret 
> professionnel. Si vous recevez ce message par erreur, merci d'en avertir 
> immédiatement l'expéditeur et de le détruire. L'intégrité du message ne 
> pouvant être assurée sur Internet, la responsabilité de Worldline ne pourra 
> être recherchée quant au contenu de ce message. Bien que les meilleurs 
> efforts soient faits pour maintenir cette transmission exempte de tout virus, 
> l'expéditeur ne donne aucune garantie à cet égard et sa responsabilité ne 
> saurait être recherchée pour tout dommage résultant d'un virus transmis.
> 
> This e-mail and the documents attached are confidential and intended solely 
> for the addressee; it may also be privileged. If you receive this e-mail in 
> error, please notify the sender immediately and destroy it. As its integrity 
> cannot be secured on the Internet, the Worldline liability cannot be 
> triggered for the message content. Although the sender endeavours to maintain 
> a computer virus-free network, the sender does not warrant that this 
> transmission is virus-free and will not be liable for any damages resulting 
> from any virus transmitted.
> 
> 
> ________________________________
> 
> Ce message et les pièces jointes sont confidentiels et réservés à l'usage 
> exclusif de ses destinataires. Il peut également être protégé par le secret 
> professionnel. Si vous recevez ce message par erreur, merci d'en avertir 
> immédiatement l'expéditeur et de le détruire. L'intégrité du message ne 
> pouvant être assurée sur Internet, la responsabilité de Worldline ne pourra 
> être recherchée quant au contenu de ce message. Bien que les meilleurs 
> efforts soient faits pour maintenir cette transmission exempte de tout virus, 
> l'expéditeur ne donne aucune garantie à cet égard et sa responsabilité ne 
> saurait être recherchée pour tout dommage résultant d'un virus transmis.
> 
> This e-mail and the documents attached are confidential and intended solely 
> for the addressee; it may also be privileged. If you receive this e-mail in 
> error, please notify the sender immediately and destroy it. As its integrity 
> cannot be secured on the Internet, the Worldline liability cannot be 
> triggered for the message content. Although the sender endeavours to maintain 
> a computer virus-free network, the sender does not warrant that this 
> transmission is virus-free and will not be liable for any damages resulting 
> from any virus transmitted.
> 
> 
> 
> Ce message et les pièces jointes sont confidentiels et réservés à l'usage 
> exclusif de ses destinataires. Il peut également être protégé par le secret 
> professionnel. Si vous recevez ce message par erreur, merci d'en avertir 
> immédiatement l'expéditeur et de le détruire. L'intégrité du message ne 
> pouvant être assurée sur Internet, la responsabilité de Worldline ne pourra 
> être recherchée quant au contenu de ce message. Bien que les meilleurs 
> efforts soient faits pour maintenir cette transmission exempte de tout virus, 
> l'expéditeur ne donne aucune garantie à cet égard et sa responsabilité ne 
> saurait être recherchée pour tout dommage résultant d'un virus transmis.
> 
> This e-mail and the documents attached are confidential and intended solely 
> for the addressee; it may also be privileged. If you receive this e-mail in 
> error, please notify the sender immediately and destroy it. As its integrity 
> cannot be secured on the Internet, the Worldline liability cannot be 
> triggered for the message content. Although the sender endeavours to maintain 
> a computer virus-free network, the sender does not warrant that this 
> transmission is virus-free and will not be liable for any damages resulting 
> from any virus transmitted.

Reply via email to