[ https://issues.apache.org/jira/browse/CASSANDRA-10449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14961726#comment-14961726 ]
Mikhail Stepura edited comment on CASSANDRA-10449 at 10/17/15 4:50 AM: ----------------------------------------------------------------------- So, there are 2 ColumnFamilyStores with 6G of memtables each (5797 'live' memtables per intsance) in the heap, but for some reason they are not being flushed. All 32 MemtableFlushWriters (and 1 post-flush) are waiting on {{writeBarrier}} s was (Author: mishail): So, there are 2 6G memtables in the heap, but for some reason they are not being flushed. All 32 MemtableFlushWriters (and 1 post-flush) are waiting on {{writeBarrier}} s > OOM on bootstrap after long GC pause > ------------------------------------ > > Key: CASSANDRA-10449 > URL: https://issues.apache.org/jira/browse/CASSANDRA-10449 > Project: Cassandra > Issue Type: Bug > Components: Core > Environment: Ubuntu 14.04, AWS > Reporter: Robbie Strickland > Labels: gc > Fix For: 2.1.x > > Attachments: GCpath.txt, heap_dump.png, system.log.10-05, > thread_dump.log, threads.txt > > > I have a 20-node cluster (i2.4xlarge) with vnodes (default of 256) and > 500-700GB per node. SSTable counts are <10 per table. I am attempting to > provision additional nodes, but bootstrapping OOMs every time after about 10 > hours with a sudden long GC pause: > {noformat} > INFO [Service Thread] 2015-10-05 23:33:33,373 GCInspector.java:252 - G1 Old > Generation GC in 1586126ms. G1 Old Gen: 49213756976 -> 49072277176; > ... > ERROR [MemtableFlushWriter:454] 2015-10-05 23:33:33,380 > CassandraDaemon.java:223 - Exception in thread > Thread[MemtableFlushWriter:454,5,main] > java.lang.OutOfMemoryError: Java heap space > {noformat} > I have tried increasing max heap to 48G just to get through the bootstrap, to > no avail. -- This message was sent by Atlassian JIRA (v6.3.4#6332)