[ https://issues.apache.org/jira/browse/CASSANDRA-6541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14341346#comment-14341346 ]
Rekha Joshi edited comment on CASSANDRA-6541 at 2/28/15 11:08 PM: ------------------------------------------------------------------ On DSE 4.5.1, Java 1.7, linux ami on aws encountered long running GC before dse was shutdown.This was a ring of 6 nodes, and no unexpected processes /load were run.The error log attached. I suspect to have hit that hotspot/java GC issue.But in this case the class unloading is enabled. JVM_OPTS="$JVM_OPTS -XX:+CMSClassUnloadingEnabled” XX:CMSInitiatingOccupancyFraction=75 Now configured to have MAX_HEAP_SIZE=“8G”, HEAP_NEWSIZE=“2G” and GC logging enabled and is stable.However if there is a heap leak, it needs a better fix. Suggestion: If the CPU utilization is going up, or heap size leak/long running GC is sensed, the nodetool -h localhost flush can auto kick in? Thanks Rekha was (Author: rekhajoshm): On DSE 4.5.1, Java 1.7, linux ami on aws encountered long running GC before dse was shutdown.This was a ring of 6 nodes, and no unexpected processes /load were run.The error log attached. I suspect to have hit that hotspot/java GC issue.But in this case the class unloading is enabled. JVM_OPTS="$JVM_OPTS -XX:+CMSClassUnloadingEnabled” XX:CMSInitiatingOccupancyFraction=75 Now configured to have MAX_HEAP_SIZE=“8G”, HEAP_NEWSIZE=“2G” and GC logging enabled and is stable.However if there is a heap leak, it needs a better fix. Thanks Rekha > New versions of Hotspot create new Class objects on every JMX connection > causing the heap to fill up with them if CMSClassUnloadingEnabled isn't set. > ----------------------------------------------------------------------------------------------------------------------------------------------------- > > Key: CASSANDRA-6541 > URL: https://issues.apache.org/jira/browse/CASSANDRA-6541 > Project: Cassandra > Issue Type: Bug > Components: Config > Reporter: jonathan lacefield > Assignee: Brandon Williams > Priority: Minor > Fix For: 1.2.16, 2.0.6, 2.1 beta2 > > Attachments: dse_systemlog > > > Newer versions of Oracle's Hotspot JVM , post 6u43 (maybe earlier) and 7u25 > (maybe earlier), are experiencing issues with GC and JMX where heap slowly > fills up overtime until OOM or a full GC event occurs, specifically when CMS > is leveraged. Adding: > {noformat} > JVM_OPTS="$JVM_OPTS -XX:+CMSClassUnloadingEnabled" > {noformat} > The the options in cassandra-env.sh alleviates the problem. -- This message was sent by Atlassian JIRA (v6.3.4#6332)