[
https://issues.apache.org/jira/browse/LUCENE-8932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16923137#comment-16923137
]
Adrien Grand commented on LUCENE-8932:
--------------------------------------
I was hoping LUCENE-8833 would land and we could build on it but it's taking
some more time than I thought yet I don't think we need to wait. I'd suggest
the following to move forward:
- Add a parameter to BKDReader's constructor to decide whether the index
should be on or off heap. This is useful so that tests can use both flavors of
index.
- Keep a constructor that only takes an IndexInput, and load the index off
heap iff the IndexInput extends ByteBufferIndexInput. This would be consistent
with the terms index.
- Update TestBKD to randomly load the index on or off-heap.
Then when LUCENE-8833 lands we can look into doing this a bit differently.
[~jdconradson] Does it sound good to you?
The patch looks good to me in general. One minor thing that looks inconsistent
to me is that BKDInput#setPosition doesn't throw an IOException. It forces the
offheap impl to catch and rethrow, while readBytes throws an IOException so
it's up to callers to deal with the exception. In my opinion, we should make
setPosition and readBytes consistent: either both throw an IOException, or none
of them do?
> Allow BKDReader packedIndex to be off heap
> ------------------------------------------
>
> Key: LUCENE-8932
> URL: https://issues.apache.org/jira/browse/LUCENE-8932
> Project: Lucene - Core
> Issue Type: Improvement
> Reporter: Jack Conradson
> Priority: Minor
> Attachments: LUCENE-8932.patch
>
>
> This change modifies BKDReader to read the packedIndex bytes off heap rather
> than load them all on heap at a single time.
> Questions for discussion:
> # Should BKDReader only support packedIndex off heap?
> # If not, how should the choice be made?
> Using luceneutils IndexAndSearchOpenStreetMaps present the following test
> results:
> with -box -points (patch)
> READER MB: 1.1345596313476562
> BEST M hits/sec: 73.34277344984474
> BEST QPS: 74.63011169783009
> with -box -points (original)
> READER MB: 1.7249317169189453
> BEST M hits/sec: 73.77125157623486
> BEST QPS: 75.06611062353801
> with -nearest 10 -points (patch)
> READER MB: 1.1345596313476562
> BEST M hits/sec: 0.013586298373879497
> BEST QPS: 1358.6298373879497
> with -nearest 10 -points (original)
> READER MB: 1.7249317169189453
> BEST M hits/sec: 0.01445208197367343
> BEST QPS: 1445.208197367343
> with -box -geo3d (patch)
> READER MB: 1.1345596313476562
> BEST M hits/sec: 39.84968715299074
> BEST QPS: 40.54914292796736
> with -box -geo3d (original)
> READER MB: 1.7456226348876953
> BEST M hits/sec: 40.45051734329004
> BEST QPS: 41.160519101846695
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
