Hi there,
I'm running a cassandra 0.8.6 cluster with 2 nodes (in 2 DC's), RF = 2.
Actual data on the nodes is only 1GB. Disk latency < 1ms. Disk throughput ~
0.4MB/s. OS load always below 1 (on a 8 core machine with 16GB ram).
When I'm running my writes against the cluster with cl = ONE all reads
What can you see in vmstat/dstat ?
Le 5 janv. 2012 11:58, "R. Verlangen" a écrit :
> Hi there,
>
> I'm running a cassandra 0.8.6 cluster with 2 nodes (in 2 DC's), RF = 2.
> Actual data on the nodes is only 1GB. Disk latency < 1ms. Disk throughput ~
> 0.4MB/s. OS load always below 1 (on a 8 core m
CPU is idle (< 10% usage). Disk reads occasionally blocks over 32/64K.
Writes around 0-5MB per second. Network traffic 0.1 / 0.1 MB/s (in / out).
Paging 0. System int ~ 1300, csw ~ 2500.
2012/1/5 Philippe
> What can you see in vmstat/dstat ?
> Le 5 janv. 2012 11:58, "R. Verlangen" a écrit :
>
>
As I posted this I noticed that the other node's CPU is running high on
some other cronjobs (every couple of minutes to 60% usage). Is the lack of
more CPU cycles a problem in this case?
Robin
2012/1/5 R. Verlangen
> CPU is idle (< 10% usage). Disk reads occasionally blocks over 32/64K.
> Write
Depending on the CL you're reading at it will yes : if the CL requires that
the "slow" node create a digest of the data and send it to the coordinator
then it might explain the poor performance on reads. What is your read CL ?
2012/1/5 R. Verlangen
> As I posted this I noticed that the other nod
I'm also reading with CL = ONE
2012/1/5 Philippe
> Depending on the CL you're reading at it will yes : if the CL requires
> that the "slow" node create a digest of the data and send it to the
> coordinator then it might explain the poor performance on reads. What is
> your read CL ?
>
> 2012/1/5
What if you shutdown the cassandra service on the slow node, does that
improve your read performance ?
If it does then that sole node is responsible for the slow down because it
can't act as a coordinator fast enough.
2012/1/5 R. Verlangen
> I'm also reading with CL = ONE
>
>
> 2012/1/5 Philippe
It does not appear to affect the response time, certainly not in a positive
way.
2012/1/5 Philippe
> What if you shutdown the cassandra service on the slow node, does that
> improve your read performance ?
> If it does then that sole node is responsible for the slow down because it
> can't act a
You may be overloading the cluster though...
My hypothesis is that your traffic is being spread across your node and
that one slow node is slowing down the fraction of traffic that goes to
that node (when it's acting as coordinator).
So what I would do is reduce the read load a lot to make sure I
The write and read load is very minimal the moment. Roughly 10 writes + 10
reads / second. So 20 operations per second. Don't think that overloads my
cluster, does it?
2012/1/5 Philippe
> You may be overloading the cluster though...
>
> My hypothesis is that your traffic is being spread across y
Unless you are doing huge batches no... don't have any other idea for now...
2012/1/5 R. Verlangen
> The write and read load is very minimal the moment. Roughly 10 writes + 10
> reads / second. So 20 operations per second. Don't think that overloads my
> cluster, does it?
>
>
> 2012/1/5 Philippe
What happens when you turn off the cron jobs ?
Cheers
-
Aaron Morton
Freelance Developer
@aaronmorton
http://www.thelastpickle.com
On 6/01/2012, at 6:57 AM, Philippe wrote:
> Unless you are doing huge batches no... don't have any other idea for now...
>
> 2012/1/5 R. Verlangen
I turned off 1 large cronjob which caused the CPU not to get used for ~ 60%
once every 10 minutes. Both write and read are fast now. Just think I was
overloading the node.
Weird though that shutting down the node did not improve the speed.
Thank you all for your time!
Robin
2012/1/5 aaron morto
13 matches
Mail list logo
