Re: Adding more nodes causes performance problem

[Dominic Letz]

Can you copy and example of your read and write queries? Are they both
degrading in the same way performance wise?

On Mon, Feb 9, 2015 at 8:39 PM, Laing, Michael 
wrote:

> Use token-awareness so you don't have as much coordinator overhead.
>
> ml
>
> On Mon, Feb 9, 2015 at 5:32 AM, Marcelo Valle (BLOOMBERG/ LONDON) <
> mvallemil...@bloomberg.net> wrote:
>
>> AFAIK, if you were using RF 3 in a 3 node cluster, so all your nodes had
>> all your data.
>> When the number of nodes started to grow, this assumption stopped being
>> true.
>> I think Cassandra will scale linearly from 9 nodes on, but comparing a
>> situation where all your nodes hold all your data is not really fair, as in
>> this situation Cassandra will behave as a database with two more replicas,
>> for reads.
>> I can be wrong, but this is my call.
>>
>> From: user@cassandra.apache.org
>> Subject: Re:Adding more nodes causes performance problem
>>
>> I have a cluster with 3 nodes, the only keyspace is with replication
>> factor of 3,
>> the application read/write UUID-keyed data. I use CQL (casssandra-python),
>> most writes are done by execute_async, most read are done with consistency
>> level of ONE, overall performance in this setup is better than I expected.
>>
>> Then I test 6-nodes cluster and 9-nodes. The performance (both read and
>> write) was getting worse and worse. Roughly speaking, 6-nodes is about 2~3
>> times slower than 3-nodes, and 9-nodes is about 5~6 times slower than
>> 3-nodes. All tests were done with same data set, same test program, same
>> client machines, for multiple times. I'm running Cassandra 2.1.2 with
>> default
>> configuration.
>>
>> What I observed, is that with 6-nodes and 9-nodes, the Cassandra servers
>> were doing OK with IO, but CPU utilization was about 60%~70% higher than
>> 3-nodes.
>>
>> I'd like to get suggestion how to troubleshoot this, as this is totally
>> against
>> what I read, that Cassandra is scaled linearly.
>>
>>
>>
>>
>


-- 
Dominic Letz
Director of R&D
Exosite 

Re: Adding more nodes causes performance problem

[Laing, Michael]

Use token-awareness so you don't have as much coordinator overhead.

ml

On Mon, Feb 9, 2015 at 5:32 AM, Marcelo Valle (BLOOMBERG/ LONDON) <
mvallemil...@bloomberg.net> wrote:

> AFAIK, if you were using RF 3 in a 3 node cluster, so all your nodes had
> all your data.
> When the number of nodes started to grow, this assumption stopped being
> true.
> I think Cassandra will scale linearly from 9 nodes on, but comparing a
> situation where all your nodes hold all your data is not really fair, as in
> this situation Cassandra will behave as a database with two more replicas,
> for reads.
> I can be wrong, but this is my call.
>
> From: user@cassandra.apache.org
> Subject: Re:Adding more nodes causes performance problem
>
> I have a cluster with 3 nodes, the only keyspace is with replication
> factor of 3,
> the application read/write UUID-keyed data. I use CQL (casssandra-python),
> most writes are done by execute_async, most read are done with consistency
> level of ONE, overall performance in this setup is better than I expected.
>
> Then I test 6-nodes cluster and 9-nodes. The performance (both read and
> write) was getting worse and worse. Roughly speaking, 6-nodes is about 2~3
> times slower than 3-nodes, and 9-nodes is about 5~6 times slower than
> 3-nodes. All tests were done with same data set, same test program, same
> client machines, for multiple times. I'm running Cassandra 2.1.2 with
> default
> configuration.
>
> What I observed, is that with 6-nodes and 9-nodes, the Cassandra servers
> were doing OK with IO, but CPU utilization was about 60%~70% higher than
> 3-nodes.
>
> I'd like to get suggestion how to troubleshoot this, as this is totally
> against
> what I read, that Cassandra is scaled linearly.
>
>
>
>