GitHub user tboeghk opened a pull request:
https://github.com/apache/lucene-solr/pull/198
[SOLR-10550] Improve FileFloatSource eviction // reduce FileFloatSource
memory footprint
As a follow up from `SOLR-10506` we found another possible memory leak in
Solr. The values generated from an `ExternalFileField` are cached in a static
cache inside the `FileFloatSource`. That cache caches both a `IndexReader` and
`FileFloatSource`s loaded using that `IndexReader`.
Cache eviction is left to the internally used WeakHashMap or a full
eviction can be triggered via url. We are dealing with large synonym files and
word lists stored in managed resources. Those are tied to the SolrCore as
described in `SOLR-10506`. We're also using `ExternalFileField`s whose
`FileFloatSource` are cached in said static cache. The FileFloatSource hold
strong (transitive) references to the SolrCore they have been created for.
After a couple of collection reloads, the cache eviction mechanism of the
`WeakHashMap` gets activated pretty close to heap exhaustion. The patch
attached adds a mechanism to evict cache entries created in the context of a
`SolrCore` upon it's close using a close hook in the
`ExternalFileFieldReloader`. It furthermore adds a static cache reset method
for all entries bound to a given IndexReader. I'm not sure, if the added cache
resets are too aggressive or executed too often, I'd like to leave that to the
experts.
N.B.: I did this second PR for the same issue to separate code changes for
both SOLR-10506 and SOLR-10550 which I maintained on the same fork branch :-/
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/shopping24/lucene-solr branch_6_5__SOLR-10550
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/lucene-solr/pull/198.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #198
----
commit 024e950f840b8466272014e4594e198560760c79
Author: Torsten Bøgh KoÌster <[email protected]>
Date: 2017-04-21T12:51:21Z
clear cached field sources on core close
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
