selvaganesang commented on code in PR #1229:
URL: https://github.com/apache/arrow-java/pull/1229#discussion_r3707181924
##########
vector/src/main/java/org/apache/arrow/vector/complex/ListVector.java:
##########
@@ -263,18 +263,33 @@ public void exportCDataBuffers(List<ArrowBuf> buffers,
ArrowBuf buffersPtr, long
/** Set the reader and writer indexes for the inner buffers. */
private void setReaderAndWriterIndex() {
+ final long requiredOffsetBufferCapacity = (long) (valueCount + 1) *
OFFSET_WIDTH;
validityBuffer.readerIndex(0);
offsetBuffer.readerIndex(0);
if (valueCount == 0) {
validityBuffer.writerIndex(0);
+ ensureEmptyOffsetBufferCapacity(requiredOffsetBufferCapacity);
} else {
validityBuffer.writerIndex(BitVectorHelper.getValidityBufferSizeFromCount(valueCount));
}
// IPC serializer will determine readable bytes based on `readerIndex` and
`writerIndex`.
// Both are set to 0 means 0 bytes are written to the IPC stream which
will crash IPC readers
// in other libraries. According to Arrow spec, we should still output the
offset buffer which
// is [0].
- offsetBuffer.writerIndex((long) (valueCount + 1) * OFFSET_WIDTH);
+ offsetBuffer.writerIndex(requiredOffsetBufferCapacity);
+ }
+
+ private void ensureEmptyOffsetBufferCapacity(long requiredCapacity) {
+ if (offsetBuffer.capacity() >= requiredCapacity) {
+ return;
+ }
+ long previousOffsetAllocationSizeInBytes = offsetAllocationSizeInBytes;
+ ArrowBuf oldOffsetBuffer = offsetBuffer;
+ offsetBuffer = allocateOffsetBuffer(requiredCapacity);
+ offsetBuffer.setBytes(
+ 0, oldOffsetBuffer, 0, Math.min(oldOffsetBuffer.capacity(),
requiredCapacity));
Review Comment:
If requiredCapacity exceeds oldOffsetBuffer.capacity(), the remaining offset
entries cannot be left uninitialized or zeroed — per the Arrow spec the offset
buffer must be monotonically non-decreasing, so unfilled entries must be set to
the last copied offset value to represent empty lists at those positions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
