[
https://issues.apache.org/jira/browse/SOLR-13315?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alan Woodward reassigned SOLR-13315:
------------------------------------
Assignee: Alan Woodward
> Possible SolrIndexSearcher leak through LogWatcher and FunctionScoreQuery
> -------------------------------------------------------------------------
>
> Key: SOLR-13315
> URL: https://issues.apache.org/jira/browse/SOLR-13315
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
> Affects Versions: 7.5
> Reporter: Yury Pakhomov
> Assignee: Alan Woodward
> Priority: Major
> Attachments: SOLR-13315.patch, path_to_gc_root_from_heap_dump.png
>
>
> Here is possible leak of SolrIndexSearcher. Which prevents unused searchers
> to be reclaimed by gc.
> This problem was found after analyzing heap dump which was created before
> Full GC.
> 1) Where unused ref to SolrIndexSearcher is stored.
> Log4j2Watcher implements LogWatcher<LogEvent>
> and has CircularList<LogEvent> history inherited from LogWatcher<E>
> In history we can store Log4jLogEvent which can hold ref to
> ParameterizedMessage
> and ParameterizedMessage stores refs to all arguments of event log. (here we
> can store objects which are no longer in use directly or indirectly)
> 2) How SolrIndexSearcher can be indirectly reached through this log buffer.
> If during FunctionScoreQuery execution ExitingReaderException("The request
> took too long to iterate over terms. Timeout: " ..) will be thrown this query
> will be logged with warn level and it ref will be store in Log4j2Watcher.
> (Here can be any exception which will log this query to Log4j2Watcher)
> In general it should be ok but in this case FunctionScoreQuery indirectly
> stores ref to SolrIndexSearcher.
> As the result we have refs to already closed searchers which are no longer in
> use.
> Searcher has refs to caches (Docs, Filers, results ...) and they can not be
> reclaimed by gc.
> 3) How SolrIndexSearcher can be accessed through FunctionScoreQuery
> There is a FunctionScoreQuery which can hold ref to
> MultiplicativeBoostValuesSource
> which can hold ref to WrappedDoubleValuesSource
> and the last one can hold ref to SolrIndexSearcher.
> public final class FunctionScoreQuery extends Query
> { ... private final DoubleValuesSource source; ... }
> private static class MultiplicativeBoostValuesSource extends
> DoubleValuesSource
> { private final DoubleValuesSource boost; ... }
> private static class WrappedDoubleValuesSource extends DoubleValuesSource
> { private final ValueSource in; private IndexSearcher searcher; ... }
> Actually any DoubleValuesSource implementation which stores ref to
> IndexSearcher
> on method call
> public abstract DoubleValuesSource rewrite(IndexSearcher reader) throws
> IOException;
> can couse a problem if it will be logged via Log4j2Watcher.
> 4) How to temporary solve this problem
> It is possible to disable Log4j2Watcher in solr.xml
> 5) How to fix this issue in more reliable way ?
> I think that it is very dangerous to buffer refs to log messages arguments.
> And may be Log4j2Watcher should be reworked to avoid buffering refs but
> LoggingHandler depends on Log4j2Watcher.
> But may be there are better ways to solve this issue.
> Path to gc root is attached.
> !path_to_gc_root_from_heap_dump.png|width=811,height=387!
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
