zhuqi-lucas commented on code in PR #10702:
URL: https://github.com/apache/arrow-rs/pull/10702#discussion_r3792041060
##########
parquet/src/arrow/async_reader/mod.rs:
##########
@@ -654,10 +654,9 @@ impl<T: AsyncFileReader + Send + 'static>
ParquetRecordBatchStreamBuilder<T> {
schema,
fields,
batch_size,
- row_groups,
+ row_group_plan,
Review Comment:
async `build` threads `row_group_plan` through but never calls
`into_global()` — what happens if `with_row_group_selections` reaches here? A
one-line test pinning either "works" or "clean error" would prevent a silent
misbehave.
##########
parquet/src/arrow/push_decoder/mod.rs:
##########
@@ -249,6 +250,45 @@ impl ParquetPushDecoderBuilder {
}
}
+ /// Select row groups and rows using row-group-local coordinates.
+ ///
+ /// Entries are decoded in the supplied order, and omitted row groups are
+ /// skipped. A row group listed more than once is decoded once per entry.
+ /// A `None` selection reads the entire row group. A selection shorter
+ /// than its row group skips the trailing rows, while a selection longer
+ /// than its row group returns an error from [`Self::build`]. Each
+ /// selection retains its existing bitmap or selector representation.
+ ///
+ /// This configuration is mutually exclusive with
+ /// [`ArrowReaderBuilder::with_row_groups`] and
+ /// [`ArrowReaderBuilder::with_row_selection`]. Combining them returns an
+ /// error from [`Self::build`]. Calling this method more than once replaces
+ /// the previous row-group-local configuration.
+ ///
+ /// ```no_run
+ /// # use parquet::arrow::arrow_reader::RowSelection;
+ /// # use parquet::arrow::push_decoder::{ParquetPushDecoderBuilder,
RowGroupSelection};
+ /// # fn configure(
+ /// # builder: ParquetPushDecoderBuilder,
+ /// # bitmap_selection: RowSelection,
+ /// # selector_selection: RowSelection,
+ /// # ) -> ParquetPushDecoderBuilder {
+ /// builder.with_row_group_selections(vec![
+ /// RowGroupSelection::new(0, Some(bitmap_selection)),
+ /// RowGroupSelection::new(2, Some(selector_selection)),
+ /// RowGroupSelection::new(3, None),
+ /// ])
+ /// # }
+ /// ```
+ pub fn with_row_group_selections(
Review Comment:
Worth adding tests while fresh: `PerRowGroup` + offset/limit, a duplicate RG
index (doc says "decoded once per entry"), a selection shorter than its RG
(trailing-skip), and empty `vec![]` (currently reads nothing).
##########
parquet/src/arrow/push_decoder/remaining.rs:
##########
@@ -142,35 +223,54 @@ impl RowGroupFrontier {
/// Advance queued row groups until one should be handed to the builder.
fn next_readable_row_group(&mut self) -> Result<Option<NextRowGroup>,
ParquetError> {
loop {
- let Some(&row_group_idx) = self.row_groups.front() else {
+ let Some(row_group_idx) = self.queued.front() else {
return Ok(None);
};
if self.budget.is_exhausted()
Review Comment:
nit: this early-exit only fires for `Global`; `PerRowGroup` drains via the
`selected_rows == 0 → continue` path below — a short comment on that asymmetry
would help.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
