etseidl commented on code in PR #11157:
URL: https://github.com/apache/arrow-rs/pull/11157#discussion_r4151608916
##########
parquet/src/arrow/arrow_reader/mod.rs:
##########
@@ -774,6 +776,47 @@ impl ArrowReaderOptions {
self
}
+ /// Sets the same [`ColumnChunkMask`] for both page-index structures.
+ pub fn with_page_index_mask(self, mask: ColumnChunkMask) -> Self {
+ self.with_column_index_mask(mask.clone())
+ .with_offset_index_mask(mask)
+ }
+
+ /// Sets the [`ColumnChunkMask`] for the Parquet [ColumnIndex] structure.
+ ///
+ /// The column index can be costly to decode and store, especially when it
is needed
+ /// only for a subset of row groups or columns (such as when filtering by
a predicate
+ /// on a single column). Providing a [`ColumnChunkMask`] can greatly
decrease
+ /// the time needed to decode this metadata.
+ ///
+ /// The mask applies only if the column-index policy is not
[`PageIndexPolicy::Skip`]
+ /// (the default), or an underlying reader is configured to preload the
index. It is
+ /// honored by loading APIs such as [`ArrowReaderMetadata::load`];
+ /// [`ArrowReaderMetadata::try_new`] does not load or filter page indexes.
+ ///
+ /// [ColumnIndex]:
https://github.com/apache/parquet-format/blob/master/PageIndex.md
Review Comment:
The origin story is that back in the day, the column index had a `NONE`
placeholder variant, so a missing column index was replaced with that. The
offset index had no such placeholder, as the belief was all offset indexes
would be populated (which makes sense, you can't do page pruning without them).
The offset index parser simply panic'd if an individual cell was missing. To
get rid of the panic, the policy was introduced; if the policy was `Required`,
then an error was returned if an index was missing, if `Optional` then the
entire offset index would be thrown away and decoding stopped. Now that
individual cells of the offset index can be `None`, there's less need for
`Required`. I think we should at least change the policy to `Optional` when the
old `with_page_index(true)` is used.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
