Re: Cassandra nodes loaded unequally
After some time. I believe this is correct. The load seems to be correlated to compactions/number of files for keyspace/IO etc. Thanks all! Regards, On Thu, Oct 18, 2012 at 9:35 PM, aaron morton aa...@thelastpickle.comwrote: At times of high load check the CPU % for the java service running C* to confirm C* is the source of load. If the load is generated from C* check the logs (or use OpsCentre / other monitoring) to see if it correlated to compaction, or Garbage Collection or repair or high throughput. Cheers - Aaron Morton Freelance Developer @aaronmorton http://www.thelastpickle.com On 17/10/2012, at 12:22 PM, Ben Kaehne ben.kae...@sirca.org.au wrote: Nothing unusual. All servers are exactly the same. Nothing unusual in the log files. Is there any level of logging that I should be turning on? Regards, On Wed, Oct 17, 2012 at 9:51 AM, Andrey Ilinykh ailin...@gmail.comwrote: With your environment (3 nodes, RF=3) it is very difficult to get uneven load. Each node receives the same number of read/write requests. Probably something is wrong on low level, OS or VM. Do you see anything unusual in log files? Andrey On Tue, Oct 16, 2012 at 3:40 PM, Ben Kaehne ben.kae...@sirca.org.au wrote: Not connecting to the same node every time. Using Hector to ensure an even distribution of connections accross the cluster. Regards, On Sat, Oct 13, 2012 at 4:15 AM, B. Todd Burruss bto...@gmail.com wrote: are you connecting to the same node every time? if so, spread out your connections across the ring On Fri, Oct 12, 2012 at 1:22 AM, Alexey Zotov azo...@griddynamics.com wrote: Hi Ben, I suggest you to compare amount of queries for each node. May be the problem is on the client side. Yoy can do that using JMX: org.apache.cassandra.db:type=ColumnFamilies,keyspace=YOUR KEYSPACE,columnfamily=YOUR CF,ReadCount org.apache.cassandra.db:type=ColumnFamilies,keyspace=YOUR KEYSPACE,columnfamily=YOUR CF,WriteCount Also I suggest to check output of nodetool compactionstats. -- Alexey -- -Ben -- -Ben -- -Ben
Re: Cassandra nodes loaded unequally
I checked this and all the numbers seemed to be about the same. Although the files would compact from time to time. There was nothing to suggest why 1 node, ongoingly had less load then the others. Regards, On Fri, Oct 12, 2012 at 7:22 PM, Alexey Zotov azo...@griddynamics.comwrote: Hi Ben, I suggest you to compare amount of queries for each node. May be the problem is on the client side. Yoy can do that using JMX: org.apache.cassandra.db:type=ColumnFamilies,keyspace=YOUR KEYSPACE,columnfamily=YOUR CF,ReadCount org.apache.cassandra.db:type=ColumnFamilies,keyspace=YOUR KEYSPACE,columnfamily=YOUR CF,WriteCount Also I suggest to check output of nodetool compactionstats. -- Alexey -- -Ben
Re: Cassandra nodes loaded unequally
Not connecting to the same node every time. Using Hector to ensure an even distribution of connections accross the cluster. Regards, On Sat, Oct 13, 2012 at 4:15 AM, B. Todd Burruss bto...@gmail.com wrote: are you connecting to the same node every time? if so, spread out your connections across the ring On Fri, Oct 12, 2012 at 1:22 AM, Alexey Zotov azo...@griddynamics.com wrote: Hi Ben, I suggest you to compare amount of queries for each node. May be the problem is on the client side. Yoy can do that using JMX: org.apache.cassandra.db:type=ColumnFamilies,keyspace=YOUR KEYSPACE,columnfamily=YOUR CF,ReadCount org.apache.cassandra.db:type=ColumnFamilies,keyspace=YOUR KEYSPACE,columnfamily=YOUR CF,WriteCount Also I suggest to check output of nodetool compactionstats. -- Alexey -- -Ben
Re: Cassandra nodes loaded unequally
Nothing unusual. All servers are exactly the same. Nothing unusual in the log files. Is there any level of logging that I should be turning on? Regards, On Wed, Oct 17, 2012 at 9:51 AM, Andrey Ilinykh ailin...@gmail.com wrote: With your environment (3 nodes, RF=3) it is very difficult to get uneven load. Each node receives the same number of read/write requests. Probably something is wrong on low level, OS or VM. Do you see anything unusual in log files? Andrey On Tue, Oct 16, 2012 at 3:40 PM, Ben Kaehne ben.kae...@sirca.org.au wrote: Not connecting to the same node every time. Using Hector to ensure an even distribution of connections accross the cluster. Regards, On Sat, Oct 13, 2012 at 4:15 AM, B. Todd Burruss bto...@gmail.com wrote: are you connecting to the same node every time? if so, spread out your connections across the ring On Fri, Oct 12, 2012 at 1:22 AM, Alexey Zotov azo...@griddynamics.com wrote: Hi Ben, I suggest you to compare amount of queries for each node. May be the problem is on the client side. Yoy can do that using JMX: org.apache.cassandra.db:type=ColumnFamilies,keyspace=YOUR KEYSPACE,columnfamily=YOUR CF,ReadCount org.apache.cassandra.db:type=ColumnFamilies,keyspace=YOUR KEYSPACE,columnfamily=YOUR CF,WriteCount Also I suggest to check output of nodetool compactionstats. -- Alexey -- -Ben -- -Ben
Syncing nodes + Cassandra Data Availability
Good morning, Our application runs on a 3 node cassandra cluster with RF of 3. We use quorum operations against this cluster in hopes of garunteeing consistency. One scenario in which an issue can occur here is: Out of our 3 nodes, only 2 are up. We perform a write to say, a new key. The down node is started again, at the same time, a different node is brought offline. At this point. The data we have written above is on one node, but not the other online node. Meaning quorum reads will fail. Surely other people have encountered such issue before. We disabled hinted handoffs originally as to not have to worry about race conditions of disk space on servers filling up due to piling up handoffs. Although perhaps this may somewhat aid the situation (although from what I read, it does not completely remedy the circumstance). If so, how are you dealing with it? From what I understand a read repair (in which we have set to 1.0) will only be performed on a successful read occurs, in which will not happen here. nodetool repair seems rather slow, is manual and does not suit our situation where data has to be available apon demand. Regards, -- -Ben
Cassandra startup times
Good evening, I am interested in improving the startup time of our cassandra cluster. We have a 3 node cluster (replication factor of 3) in which our application requires quorum reads and writes to function. Each machine is well specced with 24gig of ram, 10 cores, jna enabled etc. On each server our keyspace files are so far around 90 Gb (stored on NFS although I am not seeing signs that we have much network io). This size will grow in future. Our startup time for 1 server at the moment is greater then half an hour (45 minutes to 50 minutes even) which is putting a risk factor on the resiliance of our service. I have tried version 1.09 to latest 1.12. I do not see too much system utilization while starting either. I gazed apon an article suggesting increased speed in 1.2 although when I set it up, it did not seem to be any faster at all (if not slower). I was observing what was happening during startup and I noticed (via strace), cassandra was doing lots of 8 byte reads from: /var/lib/cassandra/data/XX/YY/XXX-YYY-hc-1871-CompressionInfo.db /var/lib/cassandra/data/XX/YY/XXX-YYY-hc-1874-CompressionInfo.db Also... Is there someone I can change the 8 byte reads to something greater? 8 byte reads across NFS is terribly inefficient (and I am guessing the cause of our terribly slow startup times). Regards, -- -Ben
Multiple keyspace question
Good evening, I have read multiple keyspaces are bad before in a few discussions, but to what extent? We have some reasonably powerful machines and looking to host an additional (currently we have 1) 2 keyspaces within our cassandra cluster (of 3 nodes, using RF3). At what point does adding extra keyspaces start becoming an issue? Is there anything special we should be considering or watching out for as we implement this? I could not imagine that all cassandra users out there are running one massive keyspace, and at the same time can not imaging that all cassandra users have multiple clusters just to host different keyspaces. Regards. -- -Ben
Re: Maximum load per node
Does this max load have correlation to replication factor?
IE a 3 node cluster with rf of 3. Should i be worried at {max load} X 3 or
what people generally mention the max load is?
On Thu, Jun 7, 2012 at 10:55 PM, Filippo Diotalevi fili...@ntoklo.comwrote:
Hi,
one of latest Aaron's observation about the max load per Cassandra node
caught my attention
At ~840GB I'm probably running close
to the max load I should have on a node,
[AM] roughly 300GB to 400GB is the max load
Since we currently have a Cassandra node with roughly 330GB of data, it
looks like that's a good time for us to really understand what's that limit
in our case. Also, a (maybe old) Stackoverflow question at
http://stackoverflow.com/questions/4775388/how-much-data-per-node-in-cassandra-cluster
,
seems to suggest a higher limit per node.
Just considering the compaction issues, what are the factors we need to
account to determine the max load?
* disk space
Datastax cassandra docs state (pg 97) that a major compaction temporarily
doubles disk space usage. Is it a safe estimate to say that the Cassandra
machine needs to have roughly the same amount of free disk space as the
current load of the Cassandra node, or are there any other factor to
consider?
* RAM
Is the amount of RAM in the machine (or dedicated to the Cassandra node)
affecting in any way the speed/efficiency of the compaction process?
* Performance degradation for overloaded nodes?
What kind of performance degradation can we expect for a Cassandra node
which is overloaded? (f.i. with 500GB or more of data)
Thanks for the clarifications,
--
Filippo Diotalevi
--
-Ben
