[ https://issues.apache.org/jira/browse/LUCENE-7258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15265328#comment-15265328 ]
Adrien Grand commented on LUCENE-7258: -------------------------------------- When I said per thread, I did not mean that the pool needs to be per thread, I just wanted to highlight that if you want N concurrent requests to all use this cache, you need a cache of quite a significant size. Sorry for not being encouraging but I worked on similar caches before and I was not happy with the end result: it is sometimes hard to know when an instance can actually be recycled, it is hard to know when the cache should give memory back to the JVM (especially for a library), the fact that you often end up using oversized instances encourages to use paging but then things are slower, etc. Relying on the JVM makes things simpler. > Tune DocIdSetBuilder allocation rate > ------------------------------------ > > Key: LUCENE-7258 > URL: https://issues.apache.org/jira/browse/LUCENE-7258 > Project: Lucene - Core > Issue Type: Improvement > Components: modules/spatial > Reporter: Jeff Wartes > Attachments: > LUCENE-7258-Tune-memory-allocation-rate-for-Intersec.patch, > LUCENE-7258-Tune-memory-allocation-rate-for-Intersec.patch, > allocation_plot.jpg > > > LUCENE-7211 converted IntersectsPrefixTreeQuery to use DocIdSetBuilder, but > didn't actually reduce garbage generation for my Solr index. > Since something like 40% of my garbage (by space) is now attributed to > DocIdSetBuilder.growBuffer, I charted a few different allocation strategies > to see if I could tune things more. > See here: http://i.imgur.com/7sXLAYv.jpg > The jump-then-flatline at the right would be where DocIdSetBuilder gives up > and allocates a FixedBitSet for a 100M-doc index. (The 1M-doc index > curve/cutoff looked similar) > Perhaps unsurprisingly, the 1/8th growth factor in ArrayUtil.oversize is > terrible from an allocation standpoint if you're doing a lot of expansions, > and is especially terrible when used to build a short-lived data structure > like this one. > By the time it goes with the FBS, it's allocated around twice as much memory > for the buffer as it would have needed for just the FBS. -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org