Jiayi-Wang-db opened a new pull request, #11081: URL: https://github.com/apache/arrow-rs/pull/11081
# Which issue does this PR close? This is an experimental draft for design review and does not yet close an issue. # Rationale for this change Parquet's monolithic footer requires an object-store reader to fetch and decode all file metadata before it can plan projected reads. This prototype integrates the modular-footer format with the existing arrow-rs asynchronous reader so metadata for projected columns can be fetched incrementally while preserving the normal data-page decoding path. # What changes are included in this PR? This PR adds an experimental `ModularFooterReader`, a sparse scan-metadata abstraction, and `ParquetRecordBatchStreamBuilder::new_with_modular_footer`. It decodes the MFP1 tail, selectively fetches projected placement modules, reconstructs projected nested schemas, and hands ordinary Parquet column chunks to the existing async record-batch decoder. It also adds selective row-group statistics loading, an object-store example for S3, GCS, Azure, HTTP, and local files, and metadata benchmarks. Known limitations in this draft are that required page indexes are unsupported, statistics pruning is exposed but not yet connected to the stream builder, an embedded `ARROW:schema` hint requires the file-metadata module, and multiple dictionary offsets are represented by the first offset when adapting to the existing metadata model. # Are these changes tested? Focused tests cover MFP1 parsing, projected placement decoding, selective statistics loading, nested projected schema reconstruction, and equality between modular-footer and legacy reads through the normal async data-page decoder. `cargo fmt`, `git diff --check`, and strict Clippy for the async library and object-store example pass. An ignored integration test supports converter-produced fixtures; it was exercised with a 59 MB NYC taxi file containing 3,475,226 rows, 20 columns, and 4 row groups, and produced the same projected 128-row batch as the legacy reader. On a synthetic file with 10,000 columns, 10 row groups, and an 11,474,640-byte legacy footer, one-column modular metadata loading took approximately 2.83 ms versus 43.95 ms for the full legacy footer. A filter-planning benchmark for `SELECT c1 WHERE c0 >= 5000` took approximately 2.65 ms, fetched a 1 MiB tail plus the 160-byte `c0` statistics module, and pruned 5 of 10 row groups. These figures measure metadata planning rather than end-to-end cloud query latency. # Are there any user-facing changes? This draft adds experimental public async-reader APIs for modular-footer files. Existing Parquet reader behavior is unchanged. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
