nastra commented on code in PR #3620:
URL: https://github.com/apache/parquet-java/pull/3620#discussion_r3451650910
##########
parquet-hadoop/src/main/java/org/apache/parquet/hadoop/ParquetFileReader.java:
##########
@@ -1361,8 +1362,32 @@ private void readVectored(List<ConsecutivePartList>
allParts, ChunkListBuilder b
totalSize += len;
}
LOG.debug("Reading {} bytes of data with vectored IO in {} ranges",
totalSize, ranges.size());
- // Request a vectored read;
- f.readVectored(ranges, options.getAllocator());
+ // Wrap the allocator to track buffers allocated during the vectored read.
+ // Hadoop's vectored IO may merge ranges and return slices of merged
buffers,
+ // so the buffers returned via CompletableFuture may differ from those
originally
+ // allocated. We track the originals here to ensure they are properly
released.
+ ByteBufferAllocator baseAllocator = options.getAllocator();
+ List<ByteBuffer> allocatedBuffers = new ArrayList<>();
+ ByteBufferAllocator trackingAllocator = new ByteBufferAllocator() {
Review Comment:
makes sense, let's go with your PR then.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
