[
https://issues.apache.org/jira/browse/SOLR-17063?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Gibney resolved SOLR-17063.
-----------------------------------
Fix Version/s: 9.5
Resolution: Fixed
> Do not buffer raw log params in LogWatcher
> ------------------------------------------
>
> Key: SOLR-17063
> URL: https://issues.apache.org/jira/browse/SOLR-17063
> Project: Solr
> Issue Type: Improvement
> Affects Versions: main (10.0), 9.4
> Reporter: Michael Gibney
> Priority: Minor
> Fix For: 9.5
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> As [~mkhl] [suggested on
> SOLR-13315|https://issues.apache.org/jira/browse/SOLR-13315?focusedCommentId=16790317#comment-16790317],
> it would be possible for {{LogWatcher}}, rather than buffering all the raw
> parameters of a log message, to pre-compute the derivative {{SolrDocument}}s
> and buffer those instead.
> SOLR-13315 ended up being addressed in a different way, but
> "memory-leak-like" behavior is still possible and I think we should follow
> [~mkhl]'s suggestion. The problem we ran into recently was a huge distributed
> query, dispersed to lots of shards. One of the ShardRequests failed, causing
> the failed ShardResponse to be logged at warn level in SearchHandler:
> {code:java}
> log.warn("Shard request failed : {}", srsp);
> {code}
> The logged ShardResponse indirectly holds a reference (via ShardRequest ref)
> to all the other ShardResponses, some of which succeeded and were quite
> large. These references then just sit in LogWatcher's {{history}}
> CircularBuffer until they are overwritten by subsequent log messages (default
> WARN/ERROR level). So as long as you have fewer than 50 (default config)
> subsequent WARN/ERROR level messages, these references will be held (in
> principle forever). In our case it consumed a substantial portion of the heap.
> This issue proposes that LogWatcher should process LogEvents into
> SolrDocuments (the form they take upon retrieval) when they are initially
> added to the {{history}} buffer. The downside is some minimal amount of extra
> processing in the case where the events are never retrieved from the buffer;
> the upside is that the params don't needlessly retain heap.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org
For additional commands, e-mail: issues-h...@solr.apache.org
