sunchao opened a new pull request, #3726:
URL: https://github.com/apache/parquet-java/pull/3726
## Why are the changes needed?
Apache Parquet's Hadoop vectored-read path can fall back to ordinary reads
after
asynchronous filesystem requests have already been submitted or partially
consumed. Retrying against the same partially populated chunk builder can
duplicate selected page data and return incorrect filtered rows. Outstanding
sibling requests can also continue using a stream while fallback or cleanup
begins.
The same path requests one buffer for an entire contiguous projected range
instead of splitting requests at `parquet.read.allocation.size`, which can
cause
unexpectedly large allocations.
Vectored I/O is enabled by default in Apache Parquet 1.16 and later, so these
issues are not limited to applications that explicitly opt in.
## What changes were proposed in this PR?
- Split contiguous requested filesystem ranges at the configured allocation
limit while preserving the existing logical page and column plan.
- Keep ordinary fallback for failures detected while preparing vectored
ranges;
convert failures after the vectored submission call is attempted into
`IOException` rather than replaying reads against a partially populated
builder.
- Await already-published sibling read futures within the existing shared
timeout before propagating an asynchronous failure.
- Verify dictionary pages and V1/V2 data-page checksums incrementally across
split buffers instead of allocating another oversized contiguous buffer.
- Add focused vectored-reader and checksum regression coverage.
The current patch still has four known review points that need follow-up:
- Filesystems that align ranges for checksums can allocate more than the
requested range size, so splitting requests alone does not guarantee that
every backend allocation respects the configured cap.
- A backend that rejects vectored I/O before submitting work should retain
safe
ordinary-read fallback; the current submission boundary cannot distinguish
that case from a partially submitted failure.
- Filesystems such as S3A can block while synchronously submitting more
ranges
than their in-flight limit permits, before the existing read timeout
applies.
- A backend can start reads and then fail before publishing their futures to
the wrapper, leaving those reads invisible to the current cleanup path.
## How was this PR tested?
Previously recorded results for this commit on Java 17:
```bash
mvn -pl parquet-hadoop \
-Dtest=TestParquetFileReaderVectoredIO,TestDataPageChecksums \
-Dsurefire.failIfNoSpecifiedTests=false \
test
```
The focused suites reported **37 passing tests**, including **19
vectored-reader
tests** and **18 checksum tests**.
```bash
mvn -pl parquet-hadoop test
mvn -pl parquet-hadoop spotless:check
```
The complete module reported **748 tests, zero failures, zero errors, and 24
existing Hadoop-capability skips**. Apache RAT approved all **256** scanned
licenses, and Spotless passed. The four review points above are not covered
by
those recorded passing tests.
Closes #3719.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
