[
https://issues.apache.org/jira/browse/CASSANDRA-14922?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16720109#comment-16720109
]
Benedict commented on CASSANDRA-14922:
--------------------------------------
Do you know where the soft references originate? I wonder if there's anything
we can do to simply eliminate them.
Short of that, I'm comfortable setting the SoftRefLRUPolicyMSPerMB - the
default seems pretty strange (of 1s per MB), and probably does not seem to play
well with Metaspace, since this probably doesn't contribute to the input to the
SoftRefLRUPolicyMSPerMB. So we could have Gigabytes of free space, and judge
that we need the referent to have been unused for hours before we collect it.
It's been a while since I sperlunked in the JDK source, but it might be worth
taking a look to find out exactly what number if provides, but honestly I don't
really see a problem with setting the value to zero. The worst impact of soft
references being freed too-eagrerly is performance, it should never be
functional. We don't depend on them directly in C*, so it would only be via
the JDK. Worst case scenario, we need to investigate again at some later date
an issue with the tests.
Nice catch on the Native API also.
Overall this looks like excellent work (Though I'll leave a full review to
Alex, since he's marked himself reviewer)
> In JVM dtests need to clean up after instance shutdown
> ------------------------------------------------------
>
> Key: CASSANDRA-14922
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14922
> Project: Cassandra
> Issue Type: Bug
> Components: Testing
> Reporter: Joseph Lynch
> Assignee: Joseph Lynch
> Priority: Minor
> Attachments: AllThreadsStopped.png, ClassLoadersRetaining.png,
> Leaking_Metrics_On_Shutdown.png, MainClassRetaining.png,
> Metaspace_Actually_Collected.png, OnlyThreeRootsLeft.png,
> no_more_references.png
>
>
> Currently the unit tests are failing on circleci ([example
> one|https://circleci.com/gh/jolynch/cassandra/300#tests/containers/1],
> [example
> two|https://circleci.com/gh/rustyrazorblade/cassandra/44#tests/containers/1])
> because we use a small container (medium) for unit tests by default and the
> in JVM dtests are leaking a few hundred megabytes of memory per test right
> now. This is not a big deal because the dtest runs with the larger containers
> continue to function fine as well as local testing as the number of in JVM
> dtests is not yet high enough to cause a problem with more than 2GB of
> available heap. However we should fix the memory leak so that going forwards
> we can add more in JVM dtests without worry.
> I've been working with [~ifesdjeen] to debug, and the issue appears to be
> unreleased Table/Keyspace metrics (screenshot showing the leak attached). I
> believe that we have a few potential issues that are leading to the leaks:
> 1. The
> [{{Instance::shutdown}}|https://github.com/apache/cassandra/blob/f22fec927de7ac291266660c2f34de5b8cc1c695/test/distributed/org/apache/cassandra/distributed/Instance.java#L328-L354]
> method is not successfully cleaning up all the metrics created by the
> {{CassandraMetricsRegistry}}
> 2. The
> [{{TestCluster::close}}|https://github.com/apache/cassandra/blob/f22fec927de7ac291266660c2f34de5b8cc1c695/test/distributed/org/apache/cassandra/distributed/TestCluster.java#L283]
> method is not waiting for all the instances to finish shutting down and
> cleaning up before continuing on
> 3. I'm not sure if this is an issue assuming we clear all metrics, but
> [{{TableMetrics::release}}|https://github.com/apache/cassandra/blob/4ae229f5cd270c2b43475b3f752a7b228de260ea/src/java/org/apache/cassandra/metrics/TableMetrics.java#L951]
> does not release all the metric references (which could leak them)
> I am working on a patch which shuts down everything and assures that we do
> not leak memory.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
