etseidl commented on PR #11297:
URL: https://github.com/apache/arrow-rs/pull/11297#issuecomment-5940555794
A concrete motivation example. Assume a new function on the push decoder
`try_decode_page_index`. To keep the API changes to a minimum, this reuses
`try_new_with_metadata` which takes ownership of `metadata`. (The result from
`try_decode_page_index` could return a tuple with the metadata included, but
that's a little inelegant IMO). With the changes here, we can just cheaply
clone metadata and let the decoder throw away the clone when it's done.
Partial results could be cached and later merged with subsequent ones, and
there could be a provider wrapper around the cache to serve up the indexes as
needed.
```rust
// Read footer metadata without page indexes
let mut decoder = ParquetMetaDataPushDecoder::try_new(file_len)?
.with_page_index_policy(PageIndexPolicy::Skip);
let metadata = loop {
match decoder.try_decode() {
// state stuff ...
}
};
// Create a new decoder initialized with the previously decoded metadata
// Note we need to clone metadata since it is not returned to us by
try_decode_page_index
let mut decoder =
ParquetMetaDataPushDecoder::try_new_with_metadata(file_len,
metadata.clone())?
.with_column_index_mask(ColumnChunkMask::columns([0]))
.with_offset_index_mask(ColumnChunkMask::columns([0, 1, 4]));
// Decode the selected page indexes
let page_index = loop {
match decoder.try_decode_page_index() {
// state stuff ...
}
};
// Assemble new metadata for query. Once again we cheaply clone the metadata.
let metadata_for_query = metadata
.clone()
.into_builder()
.set_page_index(Some(Arc::new(page_index)))
.build();
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
