parthchandra commented on code in PR #999: URL: https://github.com/apache/parquet-mr/pull/999#discussion_r992841047
########## parquet-hadoop/src/main/java/org/apache/parquet/hadoop/ParquetFileReader.java: ########## @@ -1093,10 +1099,38 @@ private ColumnChunkPageReadStore internalReadFilteredRowGroup(BlockMetaData bloc } } } - // actually read all the chunks + // Vectored IO up. + + List<FileRange> ranges = new ArrayList<>(); for (ConsecutivePartList consecutiveChunks : allParts) { - consecutiveChunks.readAll(f, builder); + ranges.add(FileRange.createFileRange(consecutiveChunks.offset, (int) consecutiveChunks.length)); + } + LOG.warn("Doing vectored IO for ranges {}", ranges); + f.readVectored(ranges, ByteBuffer::allocate); Review Comment: Then it may make sense to do what the `readAllVectored` method is doing - split each `ConsecutiveParts` into contiguous sub-ranges of 8MB (configurable). Presumably when you merge ranges, you have a limit so you will get a large consecutive read but not hit the memory allocation issues. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@parquet.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org