Hi All, To trigger the leak I had a script do the following:
- Get old collection = collection an alias currently pointing to - Create new collection - Import to new collection - Move alias to from old collection to new collection - Delete old collection - Repeat In doing this I see, with visualvm, the number of instances of SolrCore increase and never decrease even after a number of manual garbage collections. In looking at a heap dump from visualvm with Eclipse Memory analyser, and pick a single SolrCore instance and trace the Path to GC Root, you can see that e.g. com.agent.instrumentation.solr.MeteredMetric has a reference to org.apache.solr.core.DirectUpdateHandler2 which has a reference to org.apache.solr.core.SolrCore, this is the leak. After decompiling the jar in com.newrelic.instrumentation.solr-jmx-7.4.0 and com.newrelic.instrumentation.solr-jmx-7.0.0 with intellij I see that org.apache.solr.metrics.SolrMetricManager.SolrMetricManager_Instrumentation which takes over the role of org.apache.solr.metrics.SolrMetricManager.SolrMetricManager allows Metrics to be registered, but not deregistered. org.apache.solr.metrics.SolrMetricManager.SolrMetricManager allows this with methos like unregisterGauges clearMetrics or clearRegistry which are used when object are closed. I think this is the cause of the leak and what I;ve reported to NR. Hope that helps others, and wondering if others have seen this too? Cheers, Dan On Wed, 13 Oct 2021 at 18:29, Carlos Grappa <[email protected]> wrote: > Same here, we've been bit by NR memory leaks across several of our apps. > In most cases, the solution was upgrading to the latest agent. > > In order to understand a bit better how this is affecting you, how did you > spot the leaks caused by NR? > Did you do a jmap or you have a complete snapshot of the heap? > > On Wed, Oct 13, 2021 at 2:23 PM Brad Belyeu <[email protected]> > wrote: > > > We do use NR and have also experienced performance degradation over time, > > but I had not gotten to the bottom of it yet. Thank you so much for this > > lead. Is there a public link to the ticket? > > > > On 10/13/21, 11:39 AM, "Dan Rosher" <[email protected]> wrote: > > > > Hi. > > > > We use newrelic to gather stats via their solr-jmx instrumentation > > implementation. I've noticed a memory leak with their implementation > > that > > allows metrics to be registered, but not deregistered (as > > MetricsManager > > does in Solr) > > > > I was wondering if anyone else uses newrelic and has noticed > > performance > > degradation over time in particular running out of heap as the old > gen > > gets > > filled up and exhausted. > > > > I've submitted a ticket to newrelic on this issue > > > > Kind regards, > > Dan > > > > >
