[ https://issues.apache.org/jira/browse/CASSANDRA-13929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16190865#comment-16190865 ]
Thomas Steinmaurer edited comment on CASSANDRA-13929 at 10/4/17 6:19 AM: ------------------------------------------------------------------------- Ok, if this sort of caching is entirely only useful during streaming, this somehow explains why I do not see any difference between the nodes here, cause they process regular business and no repair, bootstrapping etc, thus I can't comment on the performance gain with the cache (possibly this has been tested in CASSANDRA-9766 anyway). If the cache is useful for streaming, it still would definitely make sense IMHO, to make the max capacity configurable (with a somewhat useful small default value) via a system property, cassandra.yaml or whatever place is preferred here, cause ~ 2,4G+ heap usage for this sort of cache at -XmX8G does not make sense or perhaps the majority of Cassandra users don't scale out with smaller sized machines anymore. As you have mentioned *per thread*. I guess we are talking about the number of threads serving client requests, aka e.g. {{native_transport_max_threads}}? If so, there should be somewhere a clear pointer in a comment, documentation etc., that heap usage is directly related to number of threads * configurable max size was (Author: tsteinmaurer): Ok, if this sort of caching is entirely only useful during streaming, this somehow explains why I do not see any difference between the nodes here, cause they process regular business and no repair, bootstrapping etc, thus I can't comment on the performance gain with the cache (possibly this has been tested in CASSANDRA-9766 anyway). If the cache is useful for streaming, it still would definitely make sense IMHO, to make the max capacity configurable (with a somewhat useful small default value) via a system property, cassandra.yaml or whatever place is preferred here, cause ~ 2,4G+ heap usage for this sort of cache at -XmX8G does not make sense. As you have mentioned *per thread*. I guess we are talking about the number of threads serving client requests, aka e.g. {{native_transport_max_threads}}? If so, there should be somewhere a clear pointer in a comment, documentation etc., that heap usage is directly related to number of threads * configurable max size > BTree$Builder / io.netty.util.Recycler$Stack leaking memory > ----------------------------------------------------------- > > Key: CASSANDRA-13929 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13929 > Project: Cassandra > Issue Type: Bug > Components: Core > Reporter: Thomas Steinmaurer > Attachments: cassandra_3.11.0_min_memory_utilization.jpg, > cassandra_3.11.1_mat_dominator_classes_FIXED.png, > cassandra_3.11.1_mat_dominator_classes.png, > cassandra_3.11.1_snapshot_heaputilization.png > > > Different to CASSANDRA-13754, there seems to be another memory leak in > 3.11.0+ in BTree$Builder / io.netty.util.Recycler$Stack. > * heap utilization increase after upgrading to 3.11.0 => > cassandra_3.11.0_min_memory_utilization.jpg > * No difference after upgrading to 3.11.1 (snapshot build) => > cassandra_3.11.1_snapshot_heaputilization.png; thus most likely after fixing > CASSANDRA-13754, more visible now > * MAT shows io.netty.util.Recycler$Stack as top contributing class => > cassandra_3.11.1_mat_dominator_classes.png > * With -Xmx8G (CMS) and our load pattern, we have to do a rolling restart > after ~ 72 hours > Verified the following fix, namely explicitly unreferencing the > _recycleHandle_ member (making it non-final). In > _org.apache.cassandra.utils.btree.BTree.Builder.recycle()_ > {code} > public void recycle() > { > if (recycleHandle != null) > { > this.cleanup(); > builderRecycler.recycle(this, recycleHandle); > recycleHandle = null; // ADDED > } > } > {code} > Patched a single node in our loadtest cluster with this change and after ~ 10 > hours uptime, no sign of the previously offending class in MAT anymore => > cassandra_3.11.1_mat_dominator_classes_FIXED.png > Can' say if this has any other side effects etc., but I doubt. -- This message was sent by Atlassian JIRA (v6.4.14#64029) --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org