[
https://issues.apache.org/jira/browse/SOLR-8922?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15219235#comment-15219235
]
Erick Erickson commented on SOLR-8922:
--------------------------------------
I'm also wondering if there are some tricks where we could re-use memory rather
than allocate new all the time. I guess the reason I'm obsessing about this is
that anything that accounts for this much of the garbage collected seems like
it's worth the effort. This certainly seems like one of those odd places where
efficiency might trump simplicity..
I wonder if there's something we could do that would allocate some kind of
reusable pool. I'm thinking of something stupid-simple that we could use for
benchmarking (not for commit or production) to get a handle on the dimension of
the problem and how broadly it would apply...
After all, GC is one of the things we spend a _lot_ of time on when supporting
Solr.....
> DocSetCollector can allocate massive garbage on large indexes
> -------------------------------------------------------------
>
> Key: SOLR-8922
> URL: https://issues.apache.org/jira/browse/SOLR-8922
> Project: Solr
> Issue Type: Improvement
> Reporter: Jeff Wartes
> Attachments: SOLR-8922.patch
>
>
> After reaching a point of diminishing returns tuning the GC collector, I
> decided to take a look at where the garbage was coming from. To my surprise,
> it turned out that for my index and query set, almost 60% of the garbage was
> coming from this single line:
> https://github.com/apache/lucene-solr/blob/94c04237cce44cac1e40e1b8b6ee6a6addc001a5/solr/core/src/java/org/apache/solr/search/DocSetCollector.java#L49
> This is due to the simple fact that I have 86M documents in my shards.
> Allocating a scratch array big enough to track a result set 1/64th of my
> index (1.3M) is also almost certainly excessive, considering my 99.9th
> percentile hit count is less than 56k.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
