adriangb commented on code in PR #10784:
URL: https://github.com/apache/arrow-rs/pull/10784#discussion_r3835321147
##########
parquet/src/file/metadata/mod.rs:
##########
@@ -412,6 +412,176 @@ impl PageIndex {
None
}
}
+
+ /// Convert this `PageIndex` into a [`PageIndexBuilder`]
+ pub fn into_builder(self) -> PageIndexBuilder {
+ self.into()
+ }
+}
+
+/// Builder for constructing [`PageIndex`] structures
+///
+/// It supports:
+/// - Allocating space for indexes based on [`PageIndexPolicy`]
+/// - Populating column indexes for predicate columns (for page filtering)
+/// - Populating offset indexes for projected columns (for direct I/O)
+/// - Automatic conversion of empty structures to `None` to save memory
+pub struct PageIndexBuilder {
+ column_indexes: Option<Vec<Vec<Option<ColumnIndexMetaData>>>>,
+ offset_indexes: Option<Vec<Vec<Option<OffsetIndexMetaData>>>>,
Review Comment:
Won't these end up being very sparse? E.g.
```
[
[None, None, None, ..., Some(...), None, None], # row group 0, only 1
column
[None, None, None, ..., Some(...), None, None], # row group 1, only 1
column
]
```
I would ideally like DataFusion/arrow to make most allocations and
operations O(projected_cols) instead of O(schema_cols) since it's very common
for analytical datasets to be extremely wide but project very few columns for a
given query.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
