Sasilekha commented on code in PR #16452:
URL: https://github.com/apache/lucene/pull/16452#discussion_r3769746500
##########
lucene/misc/src/java/org/apache/lucene/misc/search/MemoryAccountingBitsetCollectorManager.java:
##########
@@ -47,21 +55,25 @@ public MemoryAccountingBitsetCollector newCollector() {
@Override
public Result reduce(Collection<MemoryAccountingBitsetCollector> collectors)
{
- int globalMaxDocEnd = 0;
+ // Size the result to just cover the highest matched doc across all
collectors. Each
+ // collector's maxDocEnd is inflated by doSetNextReader to the full leaf
regardless of what
+ // actually matches, so keying off it can significantly over-allocate on
selective queries or
+ // narrow intra-segment slices; use the actual high-water mark tracked at
collect time.
+ int resultSize = 0;
for (MemoryAccountingBitsetCollector collector : collectors) {
- globalMaxDocEnd = Math.max(globalMaxDocEnd, collector.getMaxDocEnd());
+ int last = collector.getHighestSetBit();
+ if (last >= 0) {
+ resultSize = Math.max(resultSize, collector.getMinDocBase() + last +
1);
+ }
}
- // TODO: with intra-segment concurrency enabled, globalMaxDocEnd equals
the full index maxDoc
- // even when only a portion of the index was searched, causing
over-allocation of the result
- // bitset.
- FixedBitSet result = new FixedBitSet(globalMaxDocEnd);
+ FixedBitSet result = new FixedBitSet(resultSize);
Review Comment:
Thanks @javanna, both catches were spot on
fix: reduce() now allocates new FixedBitSet(Math.max(1, resultSize)) so the
empty-result case is a length-1 bitset and nextSetBit(0)/ get(0) stay safe.
Tight sizing is preserved whenever any doc matched the 8 byte cost only applies
to no match reductions
Extended testResultBitSetEmptyOnNoMatches to assert nextSetBit(0) ==
NO_MORE_DOCS, get(0) == false and length() == 1
Happy to discuss further and reshape if you'd prefer a different approach
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
