etseidl opened a new pull request, #10784:
URL: https://github.com/apache/arrow-rs/pull/10784
# Which issue does this PR close?
- Contributes to #7582.
# Rationale for this change
We have long desired the ability to only parse a subset of the Parquet Page
Index. Specifically, it would be nice to only populate the column index for
columns needed for predicates, and the offset index only for those columns
being projected.
# What changes are included in this PR?
This PR introduces a new `PageIndexBuilder` that will aid with incremental
construction of the page index. This also adds a new variant to the
`PageIndexPolicy` enum to allow selecting specific columns for inclusion. Use
of this is demonstrated in a new test
```rust
let mut reader = ParquetMetaDataReader::new()
.with_column_index_policy(PageIndexPolicy::only_columns(&[0]))
.with_offset_index_policy(PageIndexPolicy::only_columns(&[0, 1,
10, 11]));
```
If we're ok with this approach, more variants can be added for row group
selection and mixed row group/column selection.
# Are these changes tested?
Yes
# Are there any user-facing changes?
Yes, this adds new variant(s) to a public enum.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
