[ https://issues.apache.org/jira/browse/SOLR-13315?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16790358#comment-16790358 ]
Alan Woodward commented on SOLR-13315: -------------------------------------- I think the error here is in WrappedDoubleValuesSource (again - this has been a pain to get right!). DVS.rewrite() is called by FunctionScoreQuery#createWeight() and the returned IndexSearcher-specific reference is not stored on the query. Most searcher-specific implementations return new objects, but WrappedDoubleValuesSource.rewrite() returns itself (after caching the Searcher reference), which is where the leak occurs. I'll prepare a patch. > Possible SolrIndexSearcher leak through LogWatcher and FunctionScoreQuery > ------------------------------------------------------------------------- > > Key: SOLR-13315 > URL: https://issues.apache.org/jira/browse/SOLR-13315 > Project: Solr > Issue Type: Bug > Security Level: Public(Default Security Level. Issues are Public) > Affects Versions: 7.5 > Reporter: Yury Pakhomov > Priority: Major > Attachments: path_to_gc_root_from_heap_dump.png > > > Here is possible leak of SolrIndexSearcher. Which prevents unused searchers > to be reclaimed by gc. > This problem was found after analyzing heap dump which was created before > Full GC. > 1) Where unused ref to SolrIndexSearcher is stored. > Log4j2Watcher implements LogWatcher<LogEvent> > and has CircularList<LogEvent> history inherited from LogWatcher<E> > In history we can store Log4jLogEvent which can hold ref to > ParameterizedMessage > and ParameterizedMessage stores refs to all arguments of event log. (here we > can store objects which are no longer in use directly or indirectly) > 2) How SolrIndexSearcher can be indirectly reached through this log buffer. > If during FunctionScoreQuery execution ExitingReaderException("The request > took too long to iterate over terms. Timeout: " ..) will be thrown this query > will be logged with warn level and it ref will be store in Log4j2Watcher. > (Here can be any exception which will log this query to Log4j2Watcher) > In general it should be ok but in this case FunctionScoreQuery indirectly > stores ref to SolrIndexSearcher. > As the result we have refs to already closed searchers which are no longer in > use. > Searcher has refs to caches (Docs, Filers, results ...) and they can not be > reclaimed by gc. > 3) How SolrIndexSearcher can be accessed through FunctionScoreQuery > There is a FunctionScoreQuery which can hold ref to > MultiplicativeBoostValuesSource > which can hold ref to WrappedDoubleValuesSource > and the last one can hold ref to SolrIndexSearcher. > public final class FunctionScoreQuery extends Query > { ... private final DoubleValuesSource source; ... } > private static class MultiplicativeBoostValuesSource extends > DoubleValuesSource > { private final DoubleValuesSource boost; ... } > private static class WrappedDoubleValuesSource extends DoubleValuesSource > { private final ValueSource in; private IndexSearcher searcher; ... } > Actually any DoubleValuesSource implementation which stores ref to > IndexSearcher > on method call > public abstract DoubleValuesSource rewrite(IndexSearcher reader) throws > IOException; > can couse a problem if it will be logged via Log4j2Watcher. > 4) How to temporary solve this problem > It is possible to disable Log4j2Watcher in solr.xml > 5) How to fix this issue in more reliable way ? > I think that it is very dangerous to buffer refs to log messages arguments. > And may be Log4j2Watcher should be reworked to avoid buffering refs but > LoggingHandler depends on Log4j2Watcher. > But may be there are better ways to solve this issue. > Path to gc root is attached. > !path_to_gc_root_from_heap_dump.png|width=811,height=387! -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org