etseidl commented on code in PR #10784:
URL: https://github.com/apache/arrow-rs/pull/10784#discussion_r3846051434
##########
parquet/src/file/metadata/mod.rs:
##########
@@ -412,6 +412,176 @@ impl PageIndex {
None
}
}
+
+ /// Convert this `PageIndex` into a [`PageIndexBuilder`]
+ pub fn into_builder(self) -> PageIndexBuilder {
+ self.into()
+ }
+}
+
+/// Builder for constructing [`PageIndex`] structures
+///
+/// It supports:
+/// - Allocating space for indexes based on [`PageIndexPolicy`]
+/// - Populating column indexes for predicate columns (for page filtering)
+/// - Populating offset indexes for projected columns (for direct I/O)
+/// - Automatic conversion of empty structures to `None` to save memory
+pub struct PageIndexBuilder {
+ column_indexes: Option<Vec<Vec<Option<ColumnIndexMetaData>>>>,
+ offset_indexes: Option<Vec<Vec<Option<OffsetIndexMetaData>>>>,
Review Comment:
Thanks, that's a neat idea. I like how `Keep` allows for the dense case. It
would be interesting to see at what density it's better to just set `kept` to
`None`.
There was also a suggestion to hide the page index behind a trait
(https://github.com/apache/datafusion/issues/24288#issuecomment-5278689015).
This would allow for multiple implementations for sparse vs dense. We could
also have a provider that caches the raw bytes for the page index and then
parses on demand (although I don't know how much complexity we want to add to
this).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
