jpountz commented on a change in pull request #7:
URL: https://github.com/apache/lucene/pull/7#discussion_r751974577
##########
File path: lucene/core/src/java/org/apache/lucene/util/bkd/BKDWriter.java
##########
@@ -328,31 +288,97 @@ public Relation compare(byte[] minPackedValue, byte[]
maxPackedValue) {
// Not deleted!
docID = mappedDocID;
System.arraycopy(
- packedValues,
- index * bkd.config.packedBytesLength,
- state.scratchDataPackedValue,
+ mergeIntersectsVisitor.packedValues,
+ index * packedBytesLength,
+ packedValue,
0,
- bkd.config.packedBytesLength);
+ packedBytesLength);
return true;
}
}
}
+
+ private boolean collectNextLeaf() throws IOException {
+ assert pointTree.moveToChild() == false;
+ mergeIntersectsVisitor.reset();
+ do {
+ if (pointTree.moveToSibling()) {
+ // move to first child of this node and collect docs
+ while (pointTree.moveToChild()) {}
+ pointTree.visitDocValues(mergeIntersectsVisitor);
+ return true;
+ }
+ } while (pointTree.moveToParent());
+ return false;
+ }
+ }
+
+ private static class MergeIntersectsVisitor implements IntersectVisitor {
+
+ int docsInBlock = 0;
+ byte[] packedValues;
+ int[] docIDs;
+ private final int packedBytesLength;
+
+ MergeIntersectsVisitor(int packedBytesLength) {
+ this.docIDs = new int[0];
+ this.packedValues = new byte[0];
+ this.packedBytesLength = packedBytesLength;
+ }
+
+ void reset() {
+ docsInBlock = 0;
+ }
+
+ @Override
+ public void grow(int count) {
+ assert docsInBlock == 0;
+ if (docIDs.length < count) {
+ docIDs = ArrayUtil.grow(docIDs, count);
+ int packedValuesSize = Math.toIntExact(docIDs.length * (long)
packedBytesLength);
+ if (packedValuesSize > ArrayUtil.MAX_ARRAY_LENGTH) {
Review comment:
This looks problematic: users that have one value whose cardinality is
greater than `MAX_ARRAY_LENGTH / packedBytesLength` would run into this
exception with no way of avoiding it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
