Re: Cassandra Messages Dropped

2012-09-23 Thread Michael Theroux
Love the Mars lander analogies :) On Sep 23, 2012, at 5:39 PM, aaron morton wrote: >> To put in other words, Cassandra will lock down all tables until all pending >> flush requests fit in the pending queue. > This was the first issue I looked at in my Cassandra SF talk > http://www.datastax.com

Re: Cassandra Messages Dropped

2012-09-23 Thread aaron morton
> To put in other words, Cassandra will lock down all tables until all pending > flush requests fit in the pending queue. This was the first issue I looked at in my Cassandra SF talk http://www.datastax.com/events/cassandrasummit2012/presentations I've seen it occur more often with lots-o-second

Re: Cassandra Messages Dropped

2012-09-23 Thread Michael Theroux
There were no errors in the log (other than the messages dropped exception pasted below), and the node does recover. We have only a small number of secondary indexes (3 in the whole system). However, I went through the cassandra code, and I believe I've worked through this problem. Just to fi

Re: Cassandra Messages Dropped

2012-09-18 Thread aaron morton
Any errors in the log ? The node recovers ? Do you use secondary indexes ? If so check comments for memtable_flush_queue_size in the yaml. if this value is too low writes may back up. But I would not expect it to cause dropped messages. > nodetool info also shows we have over a gig of avail

Re: Cassandra Messages Dropped

2012-09-17 Thread Michael Theroux
Thanks for the response. We are on version 1.1.2. We don't see the MutationStage back up. The dump from the messages dropped error doesn't show a backup, but also watching "nodetool tpstats" doesn't show any backup there. nodetool info also shows we have over a gig of available memory on the

Re: Cassandra Messages Dropped

2012-09-17 Thread aaron morton
> INFO [ScheduledTasks:1] 2012-09-17 06:28:03,839 StatusLogger.java (line 72) > MemtablePostFlusher 1 5 0 > INFO [ScheduledTasks:1] 2012-09-17 06:28:03,840 StatusLogger.java (line 72) > FlushWriter 1 5 0 Looks suspiciously like