> 80 ms per request sounds high. I'm doing some guessing here, i am guessing memory usage is the problem..
* I assume you are not longer seeing excessive GC activity. * The key cache will not get used when you hit the row cache. I would disable the row cache if you have a random workload, which it looks like you do. * 500 million is a lot of keys to have on a single node. At the default index sample of every 128 keys it will have about 4 million samples, which is probably taking up a lot of memory. Is this testing a real world scenario or an abstract benchmark ? IMHO you will get more insight from testing something that resembles your application. Cheers ----------------- Aaron Morton Freelance Developer @aaronmorton http://www.thelastpickle.com On 26/05/2012, at 8:48 PM, Gurpreet Singh wrote: > Hi Aaron, > Here is the latest on this.. > i switched to a node with 6 disks and running some read tests, and i am > seeing something weird. > > setup: > 1 node, cassandra 1.0.9, 8 cpu, 16 gig RAM, 6 7200 rpm SATA data disks > striped 512 kb, commitlog mirrored. > 1 keyspace with just 1 column family > random partitioner > total number of keys: 500 million (the keys are just longs from 1 to 500 > million) > avg key size: 8 bytes > bloom filter size: 1 gig > total disk usage: 70 gigs compacted 1 sstable > mean compacted row size: 149 bytes > heap size: 8 gigs > keycache size: 2 million (takes around 2 gigs in RAM) > rowcache size: 1 million (off-heap) > memtable_total_space_mb : 2 gigs > > test: > Trying to do 5 reads per second. Each read is a multigetslice query for just > 1 key, 2 columns. > > observations: > row cache hit rate: 0.4 > key cache hit rate: 0.0 (this will increase later on as system moves to > steady state) > cfstats - 80 ms > > iostat (every 5 seconds): > > r/s : 400 > %util: 20% (all disks are at equal utilization) > await: 65-70 ms (for each disk) > svctm : 2.11 ms (for each disk) > r-kB/s - 35000 > > why this is weird is because.. > 5 reads per second is causing a latency of 80 ms per request (according to > cfstats). isnt this too high? > 35 MB/s is being read from the disk. That is again very weird. This number is > way too high, avg row size is just 149 bytes. Even index reads should not > cause this high data being read from the disk. > > what i understand is that each read request translates to 2 disk accesses > (because there is only 1 sstable). 1 for the index, 1 for the data. At such a > low reads/second, why is the latency so high? > > would appreciate help debugging this issue. > Thanks > Gurpreet > > > On Tue, May 22, 2012 at 2:46 AM, aaron morton <aa...@thelastpickle.com> wrote: > With > >> heap size = 4 gigs > > I would check for GC activity in the logs and consider setting it to 8 given > you have 16 GB. You can also check if the IO system is saturated > (http://spyced.blogspot.co.nz/2010/01/linux-performance-basics.html) Also > take a look at nodetool cfhistogram perhaps to see how many sstables are > involved. > > > I would start by looking at the latency reported on the server, then work > back to the client…. > > I may have missed it in the email but what recent latency for the CF is > reported by nodetool cfstats ? That's latency for a single request on a > single read thread. The default settings give you 32 read threads. > > If you know the latency for a single request, and you know you have 32 > concurrent read threads, you can get an idea of the max throughput for a > single node. Once you get above that throughput the latency for a request > will start to include wait time. > > It's a bit more complicated, because when you request 40 rows that turns into > 40 read tasks. So if two clients send a request for 40 rows at the same time > there will be 80 read tasks to be processed by 32 threads. > > Hope that helps. > > ----------------- > Aaron Morton > Freelance Developer > @aaronmorton > http://www.thelastpickle.com > > On 20/05/2012, at 4:10 PM, Radim Kolar wrote: > >> Dne 19.5.2012 0:09, Gurpreet Singh napsal(a): >>> Thanks Radim. >>> Radim, actually 100 reads per second is achievable even with 2 disks. >> it will become worse as rows will get fragmented. >>> But achieving them with a really low avg latency per key is the issue. >>> >>> I am wondering if anyone has played with index_interval, and how much of a >>> difference would it make to reads on reducing the index_interval. >> close to zero. but try it yourself too and post your findings. > >
