[
https://issues.apache.org/jira/browse/CASSANDRA-13929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16190865#comment-16190865
]
Thomas Steinmaurer edited comment on CASSANDRA-13929 at 10/4/17 6:18 AM:
-------------------------------------------------------------------------
Ok, if this sort of caching is entirely only useful during streaming, this
somehow explains why I do not see any difference between the nodes here, cause
they process regular business and no repair, bootstrapping etc, thus I can't
comment on the performance gain with the cache (possibly this has been tested
in CASSANDRA-9766 anyway).
If the cache is useful for streaming, it still would definitely make sense
IMHO, to make the max capacity configurable (with a somewhat useful small
default value) via a system property, cassandra.yaml or whatever place is
preferred here, cause ~ 2,4G+ heap usage for this sort of cache at -XmX8G does
not make sense.
As you have mentioned *per thread*. I guess we are talking about the number of
threads serving client requests, aka e.g. {{native_transport_max_threads}}? If
so, there should be somewhere a clear pointer in a comment, documentation etc.,
that heap usage is directly related to number of threads * configurable max size
was (Author: tsteinmaurer):
Ok, if this sort of caching is entirely only useful during streaming, this
somehow explains why I do not see any difference between the nodes here, cause
they process regular business and no repair, bootstrapping etc, thus I can't
comment on the performance gain with the cache (possibly this has been tested
in CASSANDRA-9766 anyway).
If the cache is useful for streaming, it still would definitely make sense
IMHO, to make the max capacity configurable via a system property,
cassandra.yaml or whatever place is preferred here, cause ~ 2,4G+ heap usage
for this sort of cache at -XmX8G does not make sense.
As you have mentioned *per thread*. I guess we are talking about the number of
threads serving client requests, aka e.g. {{native_transport_max_threads}}? If
so, there should be somewhere a clear pointer in a comment, documentation etc.,
that heap usage is directly related to number of threads * configurable max size
> BTree$Builder / io.netty.util.Recycler$Stack leaking memory
> -----------------------------------------------------------
>
> Key: CASSANDRA-13929
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13929
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Reporter: Thomas Steinmaurer
> Attachments: cassandra_3.11.0_min_memory_utilization.jpg,
> cassandra_3.11.1_mat_dominator_classes_FIXED.png,
> cassandra_3.11.1_mat_dominator_classes.png,
> cassandra_3.11.1_snapshot_heaputilization.png
>
>
> Different to CASSANDRA-13754, there seems to be another memory leak in
> 3.11.0+ in BTree$Builder / io.netty.util.Recycler$Stack.
> * heap utilization increase after upgrading to 3.11.0 =>
> cassandra_3.11.0_min_memory_utilization.jpg
> * No difference after upgrading to 3.11.1 (snapshot build) =>
> cassandra_3.11.1_snapshot_heaputilization.png; thus most likely after fixing
> CASSANDRA-13754, more visible now
> * MAT shows io.netty.util.Recycler$Stack as top contributing class =>
> cassandra_3.11.1_mat_dominator_classes.png
> * With -Xmx8G (CMS) and our load pattern, we have to do a rolling restart
> after ~ 72 hours
> Verified the following fix, namely explicitly unreferencing the
> _recycleHandle_ member (making it non-final). In
> _org.apache.cassandra.utils.btree.BTree.Builder.recycle()_
> {code}
> public void recycle()
> {
> if (recycleHandle != null)
> {
> this.cleanup();
> builderRecycler.recycle(this, recycleHandle);
> recycleHandle = null; // ADDED
> }
> }
> {code}
> Patched a single node in our loadtest cluster with this change and after ~ 10
> hours uptime, no sign of the previously offending class in MAT anymore =>
> cassandra_3.11.1_mat_dominator_classes_FIXED.png
> Can' say if this has any other side effects etc., but I doubt.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
