Re: Row cache and counters

2013-01-03 Thread aaron morton
The first thing I look for with timeouts like that is a flush storm causing blocking in the write path (due to the internal "switch lock"). Take a look in the logs, for a number of messages such as "enqueuing CF…" and "writing cf..". Look for a pattern of enqueuing cf messages that occur immed

Re: Row cache and counters

2013-01-03 Thread André Cruz
Does anyone see anything wrong in these settings? Anything to account for a 8s timeout during a counter increment? Thanks, André On 31/12/2012, at 14:35, André Cruz wrote: > On Dec 29, 2012, at 8:53 PM, Mohit Anchlia wrote: > >> Can you post gc settings? Also check logs and see what it says

Re: Row cache and counters

2012-12-31 Thread André Cruz
On Dec 29, 2012, at 8:53 PM, Mohit Anchlia wrote: > Can you post gc settings? Also check logs and see what it says These are the relevant jam settings: -home /usr/lib/jvm/j2re1.6-oracle/bin/../ -ea -javaagent:/usr/share/cassandra/lib/jamm-0.2.5.jar -XX:+UseThreadPriorities -XX:ThreadPriorityP

Re: Row cache and counters

2012-12-29 Thread Mohit Anchlia
Can you post gc settings? Also check logs and see what it says Also post how many writes and reads along with avg row size Sent from my iPhone On Dec 29, 2012, at 12:28 PM, rohit bhatia wrote: > i assume u mean 8 seconds and not 8ms.. > thats pretty huge to be caused by gc. Is there lot of lo

Re: Row cache and counters

2012-12-29 Thread rohit bhatia
i assume u mean 8 seconds and not 8ms.. thats pretty huge to be caused by gc. Is there lot of load on your servers? You might also need to check for memory contention Regarding GC, since its parnew all u can really do is increase heap and young gen size, or modify tenuring rate. But that can't be

Re: Row cache and counters

2012-12-29 Thread André Cruz
On 29/12/2012, at 16:59, rohit bhatia wrote: > Reads during a write still occur during a counter increment with CL ONE, but > that latency is not counted in the request latency for the write. Your local > node write latency of 45 microseconds is pretty quick. what is your timeout > and the wri

Re: Row cache and counters

2012-12-29 Thread rohit bhatia
Reads during a write still occur during a counter increment with CL ONE, but that latency is not counted in the request latency for the write. Your local node write latency of 45 microseconds is pretty quick. what is your timeout and the write request latency you see. In our deployment we had some

Row cache and counters

2012-12-29 Thread André Cruz
Hello. I recently was having some timeout issues while updating counters and turned on row cache for that particular CF. This is its stats: Column Family: UserQuotas SSTable count: 3 Space used (live): 2687239 Space used (total): 26