bruno-roustant commented on a change in pull request #2213: URL: https://github.com/apache/lucene-solr/pull/2213#discussion_r565404249
########## File path: lucene/core/src/java/org/apache/lucene/codecs/lucene80/Lucene80DocValuesConsumer.java ########## @@ -736,49 +736,92 @@ private void doAddSortedField(FieldInfo field, DocValuesProducer valuesProducer) private void addTermsDict(SortedSetDocValues values) throws IOException { final long size = values.getValueCount(); meta.writeVLong(size); - meta.writeInt(Lucene80DocValuesFormat.TERMS_DICT_BLOCK_SHIFT); + boolean compress = + Lucene80DocValuesFormat.Mode.BEST_COMPRESSION == mode Review comment: @zhaih could you be more specific about this? Could you post something in the Jira issue discussion to explain how this compression is harmful for BinaryDocValues? It needs to be more visible than just in this review. ########## File path: lucene/core/src/java/org/apache/lucene/codecs/lucene80/Lucene80DocValuesConsumer.java ########## @@ -790,6 +833,28 @@ private void addTermsDict(SortedSetDocValues values) throws IOException { writeTermsIndex(values); } + private int compressAndGetTermsDictBlockLength( + ByteArrayDataOutput bufferedOutput, LZ4.FastCompressionHashTable ht) throws IOException { + int uncompressedLength = bufferedOutput.getPosition(); + data.writeVInt(uncompressedLength); + long before = data.getFilePointer(); + LZ4.compress(termsDictBuffer, 0, uncompressedLength, data, ht); + int compressedLength = (int) (data.getFilePointer() - before); + // Block length will be used for creating buffer for decompression, one corner case is that + // compressed length might be bigger than un-compressed length, so just return the bigger one. + return Math.max(uncompressedLength, compressedLength); + } + + private ByteArrayDataOutput maybeGrowBuffer(ByteArrayDataOutput bufferedOutput, int termLength) { + int pos = bufferedOutput.getPosition(), originalLength = termsDictBuffer.length; + if (pos + termLength >= originalLength - 1) { + int targetLength = (originalLength + termLength) << 1; Review comment: The idea is to remove this line 851 (double the size) to let the ArrayUtil.oversize() inside ArrayUtil.grow() decide what is the appropriate size growth (exponential growth with less mem waste than classic x2). The following line would become something like: termsDictBuffer = ArrayUtil.grow(termsDictBuffer, pos + termLength) ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org