Re: frequent client exceptions on 0.7.0

2011-02-21 Thread Peter Schuller
> AFAIK the MemtablePostFlusher is the TP writing sstables, if it has a queue > then there is the potential for writes to block while it waits for Memtables > to be flushed. Take a look at your Memtable settings per CF, could it be that > all the Memtables are flushing at once? There is info in

Re: frequent client exceptions on 0.7.0

2011-02-20 Thread Aaron Morton
AFAIK the MemtablePostFlusher is the TP writing sstables, if it has a queue then there is the potential for writes to block while it waits for Memtables to be flushed. Take a look at your Memtable settings per CF, could it be that all the Memtables are flushing at once? There is info in the logs

Re: frequent client exceptions on 0.7.0

2011-02-18 Thread Andy Skalet
On Thu, Feb 17, 2011 at 12:22 PM, Aaron Morton wrote: > Messages been dropped means the machine node is overloaded. Look at the > thread pool stats to see which thread pools have queues. It may be IO > related, so also check the read and write latency on the CF and use iostat. > > i would try th

Re: frequent client exceptions on 0.7.0

2011-02-17 Thread Aaron Morton
t > were not worth it. I have ended up running the nodes closer to the wire and > living with an increased rate of client side exceptions and nodes going down > for short periods. > > Dan > > -Original Message- > From: Andy Skalet [mailto:aeska...@bitjug.com] >

RE: frequent client exceptions on 0.7.0

2011-02-17 Thread Dan Hendry
tions and nodes going down for short periods. Dan -Original Message- From: Andy Skalet [mailto:aeska...@bitjug.com] Sent: February-17-11 4:18 To: Peter Schuller Cc: user@cassandra.apache.org Subject: Re: frequent client exceptions on 0.7.0 On Thu, Feb 17, 2011 at 12:37 AM, Peter Schulle

Re: frequent client exceptions on 0.7.0

2011-02-17 Thread Andy Skalet
On Thu, Feb 17, 2011 at 12:37 AM, Peter Schuller wrote: > Bottom line: Check /var/log/cassandra/system.log to begin with and see > if it's reporting anything or being restarted. Thanks, Peter. In the system.log, I see quite a few of these across several machines. Everything else in the log is I

Re: frequent client exceptions on 0.7.0

2011-02-17 Thread Peter Schuller
>   raise EOFError() > EOFError [snip] > error: [Errno 104] Connection reset by peer Sounds like you either have a firewalling/networking issues that is tearing down TCP connections, or your cassandra node is dying. Have you checked the Cassandra system log? A frequent mistake is configuring mem

frequent client exceptions on 0.7.0

2011-02-16 Thread Andy Skalet
Hello, We were occasionally experiencing client exceptions with 0.6.3, so we upgraded to 0.7.0 a couple weeks ago, but unfortunately we now get more client exceptions, and more frequently. Also, occasionally nodetool ring will show a node Down even though cassandra is still running and the node w