Seems: ROW-MUTATION-STAGE 32 3349 63897493 is the clue, too many mutation requests are pending.
Yes, I also think cassandra should add a mechanism to avoid too many requests pending (in queue). When the queue is full, just reject the request from client. Seems https://issues.apache.org/jira/browse/CASSANDRA-685 is what we want. On Tue, Apr 27, 2010 at 8:16 PM, Eric Yu <suc...@gmail.com> wrote: > I wrote a script to record the tpstats output every 5 seconds. > Here is the output just before the jvm OOM: > > > Pool Name Active Pending Completed > FILEUTILS-DELETE-POOL 0 0 280 > > STREAM-STAGE 0 0 0 > RESPONSE-STAGE 0 0 245573 > > ROW-READ-STAGE 0 0 0 > LB-OPERATIONS 0 0 0 > MESSAGE-DESERIALIZER-POOL 1 14290091 65943291 > GMFD 0 0 26670 > > LB-TARGET 0 0 0 > CONSISTENCY-MANAGER 0 0 0 > ROW-MUTATION-STAGE 32 3349 63897493 > > MESSAGE-STREAMING-POOL 0 0 3 > LOAD-BALANCER-STAGE 0 0 0 > FLUSH-SORTER-POOL 0 0 0 > MEMTABLE-POST-FLUSHER 0 0 420 > FLUSH-WRITER-POOL 0 0 420 > > AE-SERVICE-STAGE 1 1 4 > HINTED-HANDOFF-POOL 0 0 52 > > > On Tue, Apr 27, 2010 at 10:53 AM, Chris Goffinet <goffi...@digg.com>wrote: > >> I'll work on doing more tests around this. In 0.5 we used a different data >> structure that required polling. But this does seem problematic. >> >> -Chris >> >> On Apr 26, 2010, at 7:04 PM, Eric Yu wrote: >> >> I have the same problem here, and I analysised the hprof file with mat, as >> you said, LinkedBlockQueue used 2.6GB. >> I think the ThreadPool of cassandra should limit the queue size. >> >> cassandra 0.6.1 >> >> java version >> $ java -version >> java version "1.6.0_20" >> Java(TM) SE Runtime Environment (build 1.6.0_20-b02) >> Java HotSpot(TM) 64-Bit Server VM (build 16.3-b01, mixed mode) >> >> iostat >> $ iostat -x -l 1 >> Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz >> avgqu-sz await svctm %util >> sda 81.00 8175.00 224.00 17.00 23984.00 2728.00 >> 221.68 1.01 1.86 0.76 18.20 >> >> tpstats, of coz, this node is still alive >> $ ./nodetool -host localhost tpstats >> Pool Name Active Pending Completed >> FILEUTILS-DELETE-POOL 0 0 1281 >> STREAM-STAGE 0 0 0 >> RESPONSE-STAGE 0 0 473617241 >> ROW-READ-STAGE 0 0 0 >> LB-OPERATIONS 0 0 0 >> MESSAGE-DESERIALIZER-POOL 0 0 718355184 >> GMFD 0 0 132509 >> LB-TARGET 0 0 0 >> CONSISTENCY-MANAGER 0 0 0 >> ROW-MUTATION-STAGE 0 0 293735704 >> MESSAGE-STREAMING-POOL 0 0 6 >> LOAD-BALANCER-STAGE 0 0 0 >> FLUSH-SORTER-POOL 0 0 0 >> MEMTABLE-POST-FLUSHER 0 0 1870 >> FLUSH-WRITER-POOL 0 0 1870 >> AE-SERVICE-STAGE 0 0 5 >> HINTED-HANDOFF-POOL 0 0 21 >> >> >> On Tue, Apr 27, 2010 at 3:32 AM, Chris Goffinet <goffi...@digg.com>wrote: >> >>> Upgrade to b20 of Sun's version of JVM. This OOM might be related to >>> LinkedBlockQueue issues that were fixed. >>> >>> -Chris >>> >>> >>> 2010/4/26 Roland Hänel <rol...@haenel.me> >>> >>>> Cassandra Version 0.6.1 >>>> OpenJDK Server VM (build 14.0-b16, mixed mode) >>>> Import speed is about 10MB/s for the full cluster; if a compaction is >>>> going on the individual node is I/O limited >>>> tpstats: caught me, didn't know this. I will set up a test and try to >>>> catch a node during the critical time. >>>> >>>> Thanks, >>>> Roland >>>> >>>> >>>> 2010/4/26 Chris Goffinet <goffi...@digg.com> >>>> >>>> Which version of Cassandra? >>>>> Which version of Java JVM are you using? >>>>> What do your I/O stats look like when bulk importing? >>>>> When you run `nodeprobe -host XXXX tpstats` is any thread pool backing >>>>> up during the import? >>>>> >>>>> -Chris >>>>> >>>>> >>>>> 2010/4/26 Roland Hänel <rol...@haenel.me> >>>>> >>>>> I have a cluster of 5 machines building a Cassandra datastore, and I >>>>>> load bulk data into this using the Java Thrift API. The first ~250GB runs >>>>>> fine, then, one of the nodes starts to throw OutOfMemory exceptions. I'm >>>>>> not >>>>>> using and row or index caches, and since I only have 5 CF's and some 2,5 >>>>>> GB >>>>>> of RAM allocated to the JVM (-Xmx2500M), in theory, that should happen. >>>>>> All >>>>>> inserts are done with consistency level ALL. >>>>>> >>>>>> I hope with this I have avoided all the 'usual dummy errors' that lead >>>>>> to OOM's. I have begun to troubleshoot the issue with JMX, however, it's >>>>>> difficult to catch the JVM in the right moment because it runs well for >>>>>> several hours before this thing happens. >>>>>> >>>>>> One thing gets to my mind, maybe one of the experts could confirm or >>>>>> reject this idea for me: is it possible that when one machine slows down >>>>>> a >>>>>> little bit (for example because a big compaction is going on), the >>>>>> memtables >>>>>> don't get flushed to disk as fast as they are building up under the >>>>>> continuing bulk import? That would result in a downward spiral, the >>>>>> system >>>>>> gets slower and slower on disk I/O, but since more and more data arrives >>>>>> over Thrift, finally OOM. >>>>>> >>>>>> I'm using the "periodic" commit log sync, maybe also this could create >>>>>> a situation where the commit log writer is too slow to catch up with the >>>>>> data intake, resulting in ever growing memory usage? >>>>>> >>>>>> Maybe these thoughts are just bullshit. Let me now if so... ;-) >>>>>> >>>>>> >>>>>> >>>>> >>>> >>> >> >> >
