[ https://issues.apache.org/jira/browse/CASSANDRA-1014?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12890471#action_12890471 ]
Daniel Kluesing commented on CASSANDRA-1014: -------------------------------------------- Just another observation: I have a big nightly import - several hundred million records that come in through the thrift interface - which caused this behavior. For this particular CF, I don't need the level of assurance the commit log provides, we ended up making the commit log configurable on a per-column family basis (with a EnableCommitLog attribute on the CF) . Inserts to that particular column family just skip the commit log. This eliminated the gc thrashing during those imports. Been doing the nightly import for well over a month, no issues. I just needed the import to work, and killing the commitlog for CFs that take heavy inserts made the problem go away. it works so I never dug any deeper. > GC storming, possible memory leak > --------------------------------- > > Key: CASSANDRA-1014 > URL: https://issues.apache.org/jira/browse/CASSANDRA-1014 > Project: Cassandra > Issue Type: Bug > Components: Core > Affects Versions: 0.6 > Environment: debian lenny amd64 OpenJDK 64-Bit Server VM (build > 1.6.0_0-b11, mixed mode) > Reporter: Brandon Williams > Assignee: Jonathan Ellis > Fix For: 0.6.4 > > Attachments: 1014-2Gheap.png, 1014-commitlog-v2.tar.gz, > 1014-table.diff, 724-0001.png, gc2.png > > > There appears to be a GC issue due to memory pressure in the 0.6 branch. You > can see this by starting the server and performing many inserts. Quickly the > jvm will consume most of its heap, and pauses for stop-the-world GC will > begin. With verbose GC turned on, this can be observed as follows: > [GC [ParNew (promotion failed): 79703K->79703K(84544K), 0.0622980 > secs][CMS[CMS-concurrent-mark: 3.678/5.031 secs] [Times: user=10.35 sys=4.22, > real=5.03 secs] > (concurrent mode failure): 944529K->492222K(963392K), 2.8264480 secs] > 990745K->492222K(1047936K), 2.8890500 secs] [Times: user=2.90 sys=0.04, > real=2.90 secs] > After enough inserts (around 75-100 million) the server will GC storm and > then OOM. > jbellis and I narrowed this down to patch 0001 in CASSANDRA-724. Switching > LBQ with ABQ made no difference, however using batch mode instead of periodic > for the commitlog does prevent the issue from occurring. The attached > screenshot shows the heap usage in jconsole first when the issue is > exhibiting, a restart, and then the same amount of inserts when it does not. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.