sunchao opened a new issue, #3719: URL: https://github.com/apache/parquet-java/issues/3719
### Describe the bug, including details regarding any error messages, version, and platform. `ParquetFileReader` can produce unsafe fallback behavior when Hadoop vectored I/O is enabled and a filesystem partially submits or completes a vectored read before raising `IllegalArgumentException` or `UnsupportedOperationException`. The current implementation catches those exceptions around `readVectored(...)` and retries every range using ordinary reads against the same `ChunkListBuilder`. If an earlier range already populated the builder, its data is appended again. For a filtered column with selected pages `P0` and `P2`, the buffered page sequence can become `[P0, P0, P2]` although the page index still describes `[P0, P2]`. Depending on the page contents, decoding can fail or silently associate the wrong page with the selected rows. Even when no page has been consumed yet, scalar fallback is unsafe once sibling asynchronous reads may still be operating on the same stream. Current upstream code: https://github.com/apache/parquet-java/blob/8e30c4cee3c7e85a8cf2133697f13138509b05b7/parquet-hadoop/src/main/java/org/apache/parquet/hadoop/ParquetFileReader.java#L1293-L1307 The same vectored path also allocates one buffer for an entire contiguous requested range instead of honoring `parquet.read.allocation.size` (8 MiB by default), unlike the ordinary read path. Large projected column chunks or filtered pages can therefore create unexpectedly large heap allocations. The current `master` branch and Apache Parquet Java 1.18.0 contain this behavior. Expected behavior: - Preserve ordinary fallback only when vectored I/O is unavailable or range preparation fails before asynchronous submission starts. - Once submission begins, fail the read safely rather than replaying scalar reads against a partially populated builder or an active stream. - Wait for already-published sibling reads before returning the original failure. - Split filesystem byte ranges to respect the configured allocation limit without changing the logical read plan or decoded results. - Add regression coverage for partial submission/completion, pending sibling futures, filtered pages, oversized columns, and checksum-enabled reads. ### Component(s) parquet-hadoop -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
