Try turning on GC logging in Cassandra-env.sh, specifically: -XX:+PrintGCApplicationStoppedTime -Xloggc:/var/log/cassandra/gc.log
Look for things like: "Total time for which application threads were stopped: 52.8795600 seconds". Anything over about a few seconds may be causing your problem. Stop the world GC is a real pain. In my cluster I was, and still am to some extent, seeing each node go 'down' about 10-30 times a day and up to a few hundred when running major compactions (by greping through the Cassandra system log). GC tuning is an art into itself but if this is your problem, try: - lower memtable flush thresholds - reduce new gen size (which is explicitly set in 0.7.1+, the -Xmn setting) - reducing CMSInitiatingOccupancyFraction from 75 to 60 or so (maybe less) - set -XX:ParallelGCThreads=<NUMBER OF CPU CORES> - set -XX:ParallelCMSThreads=<NUMBER OF CPU CORES> Again, I would recommend you do some more research into GC tuning (http://www.oracle.com/technetwork/java/javase/gc-tuning-6-140523.html is a good place to start). Most of my recommendations above will probably reduce the chance of your nodes going 'down' but may have pretty severe negative performance impacts. In my cluster, I found the measures needed to ensure the node never (or rarely, it cant be completely prevented) went down just were not worth it. I have ended up running the nodes closer to the wire and living with an increased rate of client side exceptions and nodes going down for short periods. Dan -----Original Message----- From: Andy Skalet [mailto:aeska...@bitjug.com] Sent: February-17-11 4:18 To: Peter Schuller Cc: user@cassandra.apache.org Subject: Re: frequent client exceptions on 0.7.0 On Thu, Feb 17, 2011 at 12:37 AM, Peter Schuller <peter.schul...@infidyne.com> wrote: > Bottom line: Check /var/log/cassandra/system.log to begin with and see > if it's reporting anything or being restarted. Thanks, Peter. In the system.log, I see quite a few of these across several machines. Everything else in the log is INFO level. WARN [ScheduledTasks:1] 2011-02-17 07:19:47,491 MessagingService.java (line 545) Dropped 182 READ messages in the last 5000ms WARN [ScheduledTasks:1] 2011-02-17 08:10:06,142 MessagingService.java (line 545) Dropped 31 READ messages in the last 5000ms WARN [ScheduledTasks:1] 2011-02-17 08:11:12,237 MessagingService.java (line 545) Dropped 54 READ messages in the last 5000ms WARN [ScheduledTasks:1] 2011-02-17 08:11:17,392 MessagingService.java (line 545) Dropped 487 READ messages in the last 5000ms The machines are in EC2 with firewall permission to talk to each other, so while not the most solid of network environments, at least pretty common these days. System is not going down, and cassandra process is not dying. Andy No virus found in this incoming message. Checked by AVG - www.avg.com Version: 9.0.872 / Virus Database: 271.1.1/3447 - Release Date: 02/16/11 02:34:00