Re: reads/s suddenly dropped

2012-02-22 Thread Franc Carter
On Mon, Feb 20, 2012 at 9:42 PM, Franc Carter wrote: > On Mon, Feb 20, 2012 at 12:00 PM, aaron morton wrote: > >> Aside from iostats.. >> >> nodetool cfstats will give you read and write latency for each CF. This >> is the latency for the operation on each node. Check that to see if latency >> is

Re: reads/s suddenly dropped

2012-02-20 Thread Franc Carter
On Mon, Feb 20, 2012 at 12:00 PM, aaron morton wrote: > Aside from iostats.. > > nodetool cfstats will give you read and write latency for each CF. This is > the latency for the operation on each node. Check that to see if latency is > increasing. > > Take a look at nodetool compactionstats to see

Re: reads/s suddenly dropped

2012-02-19 Thread aaron morton
Aside from iostats.. nodetool cfstats will give you read and write latency for each CF. This is the latency for the operation on each node. Check that to see if latency is increasing. Take a look at nodetool compactionstats to see if compactions are running at the same time. The IO is throttl

Re: reads/s suddenly dropped

2012-02-19 Thread Franc Carter
On Mon, Feb 20, 2012 at 4:10 AM, Philippe wrote: > Perhaps your dataset can no longer be held in memory. Check iostats > I have been flushing the keycache and dropping the linux disk caches before each to avoid testing memory reads. One possibility that I thought of is that the success keys are

Re: reads/s suddenly dropped

2012-02-19 Thread Philippe
Perhaps your dataset can no longer be held in memory. Check iostats Le 19 févr. 2012 11:24, "Franc Carter" a écrit : > > I've been testing Cassandra - primarily looking at reads/second for our > fairly data model - one unique key with a row of columns that we always > request. I've now setup the

reads/s suddenly dropped

2012-02-19 Thread Franc Carter
I've been testing Cassandra - primarily looking at reads/second for our fairly data model - one unique key with a row of columns that we always request. I've now setup the cluster with with m1.large (2 cpus 8GB) I had loaded a months worth of data in and was doing random requests as a torture test