[
https://issues.apache.org/jira/browse/LUCENE-7258?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jeff Wartes updated LUCENE-7258:
--------------------------------
Attachment: LUCENE-7258-Tune-memory-allocation-rate-for-Intersec.patch
This patch does the following:
1. Moves the FBS threshold from 1/128th to 1/256th for
IntersectsPrefixTreeQuery.
2. Changes the expansion policy to 2x when used by IntersectsPrefixTreeQuery
3. Changed the sort algorithm in DocIdSetBuilder (for ALL usages) to
InPlaceMergeSorter, since LSBRadixSorter requires allocating a new array of
size N.
4. In order to do #1 & #2, I had to add parameter support for the threshold and
expansion policies.
Justifications:
1. Since Geospatial data is typically non-uniform, a smaller threshold seemed
reasonable.
2. A more aggressive expansion policy results in less wasted allocations,
particularly for short-lived data structures.
3. This one might be controversial since it affects more than just geospatial
search, but I thought I'd see what happened if I saved the memory. I also
considered TimSort, which has a configurable memory cost, but LUCENE-5140 gave
me some pause.
> Tune DocIdSetBuilder allocation rate
> ------------------------------------
>
> Key: LUCENE-7258
> URL: https://issues.apache.org/jira/browse/LUCENE-7258
> Project: Lucene - Core
> Issue Type: Improvement
> Components: modules/spatial
> Reporter: Jeff Wartes
> Attachments:
> LUCENE-7258-Tune-memory-allocation-rate-for-Intersec.patch
>
>
> LUCENE-7211 converted IntersectsPrefixTreeQuery to use DocIdSetBuilder, but
> didn't actually reduce garbage generation for my Solr index.
> Since something like 40% of my garbage (by space) is now attributed to
> DocIdSetBuilder.growBuffer, I charted a few different allocation strategies
> to see if I could tune things more.
> See here: http://i.imgur.com/7sXLAYv.jpg
> The jump-then-flatline at the right would be where DocIdSetBuilder gives up
> and allocates a FixedBitSet for a 100M-doc index. (The 1M-doc index
> curve/cutoff looked similar)
> Perhaps unsurprisingly, the 1/8th growth factor in ArrayUtil.oversize is
> terrible from an allocation standpoint if you're doing a lot of expansions,
> and is especially terrible when used to build a short-lived data structure
> like this one.
> By the time it goes with the FBS, it's allocated around twice as much memory
> for the buffer as it would have needed for just the FBS.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
