[ https://issues.apache.org/jira/browse/CASSANDRA-13929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16374200#comment-16374200 ]
Thomas Steinmaurer commented on CASSANDRA-13929: ------------------------------------------------ The following does not include the latest patches from Feb 22, but shows last 30d on a single node (m4.2xlarge, Xmx12G, CMS) out of our 9 node loadtest environment including various tests/patches we have applied. !cassandra_heapcpu_memleak_patching_test_30d.png|width=1280! * Blue line => AVG heap utilization * Orange line => AVG CPU utilization (not really related as usually compaction is overlaying anything else most likely) Following timelines in the chart: ||Timeframe||Deployment||Comment/Result|| |Jan 25 - Feb 1|Cassandra 3.11 public + Netty 4.0.55|(!) Heap utilization increase| |Feb 1 - Feb 6|Cassandra 3.11 public + Netty 4.0.55 + limiting Netty capacity per Thread|(!) Heap utilization increase| |Feb 6 - Feb 14|Cassandra 3.11 public + Netty 4.0.55 + my recycleHandle = null patch|(/) Heap utilization stable| |Feb 14 - Feb 23|Cassandra 3.11 public + Netty 4.0.55 + *without* recycleHandle = null patch + first [~jay.zhuang] patch from Feb 13|(/) Heap utilization stable, but slightly increased to previous| Very high-level (although from the field) compared to [~jay.zhuang] tests and benchmarks, but possibly useful for a decision process, hopefully being included in 3.11.3. Thanks guys! > BTree$Builder / io.netty.util.Recycler$Stack leaking memory > ----------------------------------------------------------- > > Key: CASSANDRA-13929 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13929 > Project: Cassandra > Issue Type: Bug > Components: Core > Reporter: Thomas Steinmaurer > Assignee: Jay Zhuang > Priority: Major > Fix For: 3.11.x > > Attachments: cassandra_3.11.0_min_memory_utilization.jpg, > cassandra_3.11.1_NORECYCLE_memory_utilization.jpg, > cassandra_3.11.1_mat_dominator_classes.png, > cassandra_3.11.1_mat_dominator_classes_FIXED.png, > cassandra_3.11.1_snapshot_heaputilization.png, > cassandra_3.11.1_vs_3.11.2recyclernullingpatch.png, > cassandra_heapcpu_memleak_patching_test_30d.png, > dtest_example_80_request.png, dtest_example_80_request_fix.png, > dtest_example_heap.png, memleak_heapdump_recyclerstack.png > > > Different to CASSANDRA-13754, there seems to be another memory leak in > 3.11.0+ in BTree$Builder / io.netty.util.Recycler$Stack. > * heap utilization increase after upgrading to 3.11.0 => > cassandra_3.11.0_min_memory_utilization.jpg > * No difference after upgrading to 3.11.1 (snapshot build) => > cassandra_3.11.1_snapshot_heaputilization.png; thus most likely after fixing > CASSANDRA-13754, more visible now > * MAT shows io.netty.util.Recycler$Stack as top contributing class => > cassandra_3.11.1_mat_dominator_classes.png > * With -Xmx8G (CMS) and our load pattern, we have to do a rolling restart > after ~ 72 hours > Verified the following fix, namely explicitly unreferencing the > _recycleHandle_ member (making it non-final). In > _org.apache.cassandra.utils.btree.BTree.Builder.recycle()_ > {code} > public void recycle() > { > if (recycleHandle != null) > { > this.cleanup(); > builderRecycler.recycle(this, recycleHandle); > recycleHandle = null; // ADDED > } > } > {code} > Patched a single node in our loadtest cluster with this change and after ~ 10 > hours uptime, no sign of the previously offending class in MAT anymore => > cassandra_3.11.1_mat_dominator_classes_FIXED.png > Can' say if this has any other side effects etc., but I doubt. -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org