xiangfu0 commented on code in PR #20215:
URL: https://github.com/apache/pinot/pull/20215#discussion_r4531273638
##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/realtime/impl/invertedindex/RealtimeLuceneTextIndex.java:
##########
@@ -255,9 +259,12 @@ private MutableRoaringBitmap getPinotDocIds(IndexSearcher
indexSearcher, Mutable
IntIterator luceneDocIDIterator = luceneDocIds.getIntIterator();
MutableRoaringBitmap actualDocIDs = new MutableRoaringBitmap();
try {
+ // One stored-fields reader for all hits, so consecutive documents in a
compressed block are not decompressed
+ // again for each hit, and only the doc id field is loaded
Review Comment:
MINOR: this comment doesn't describe what Lucene 9.12 does:
- The old `IndexSearcher#doc` → `SegmentReader#document` path already reuses
a per-thread `StoredFieldsReader` (`SegmentCoreReaders.fieldsReaderLocal`).
- A non-merge `Lucene90CompressingStoredFieldsReader` decompresses the
document again on every call (`BlockState#document`), whichever reader is used.
Suggest stating the actual reason, e.g. `// Use the non-deprecated
StoredFields API (IndexSearcher#doc is removed in Lucene 10) and load only the
doc id field`. The same applies to `MultiColumnRealtimeLuceneTextIndex` and the
PR description.
##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/realtime/impl/invertedindex/RealtimeLuceneTextIndex.java:
##########
@@ -255,9 +259,12 @@ private MutableRoaringBitmap getPinotDocIds(IndexSearcher
indexSearcher, Mutable
IntIterator luceneDocIDIterator = luceneDocIds.getIntIterator();
MutableRoaringBitmap actualDocIDs = new MutableRoaringBitmap();
try {
+ // One stored-fields reader for all hits, so consecutive documents in a
compressed block are not decompressed
+ // again for each hit, and only the doc id field is loaded
+ StoredFields storedFields = indexSearcher.storedFields();
Review Comment:
FYI, perf is mixed. `storedFields()` clones one reader per leaf on every
query, while the old path reused per-thread clones. I compared the two loops on
the same reader (Lucene 9.12, JDK 25, p50):
- 100k hits on 1 leaf: 16.5 → 15.7 ms
- 200 hits over 20 leaves: 0.32 → 0.47 ms
- 200 hits over 200 leaves: 0.24 → 0.93 ms
End-to-end `getDocIds` showed no difference within about ±20% noise. Not
blocking, since this is the cost of the Lucene 10 API, but I wouldn't call it
an optimization.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]