zhuqi-lucas commented on code in PR #19557:
URL: https://github.com/apache/datafusion/pull/19557#discussion_r2652255161
##########
datafusion/datasource-parquet/src/opener.rs:
##########
@@ -148,13 +148,19 @@ impl PreparedAccessPlan {
mut self,
file_metadata: &parquet::file::metadata::ParquetMetaData,
) -> Result<Self> {
+ // Get the row group indexes before reversing
+ let row_groups_to_scan = self.row_group_indexes.clone();
+
// Reverse the row group indexes
self.row_group_indexes =
self.row_group_indexes.into_iter().rev().collect();
// If we have a row selection, reverse it to match the new row group
order
if let Some(row_selection) = self.row_selection {
- self.row_selection =
- Some(reverse_row_selection(&row_selection, file_metadata)?);
+ self.row_selection = Some(reverse_row_selection(
+ &row_selection,
+ file_metadata,
+ &row_groups_to_scan, // Pass the original (non-reversed) row
group indexes
Review Comment:
@adriangb The main change is to pass row_group_indexes to reverse logic
instead of using all indexes.
Because i saw the following logic:
https://github.com/apache/datafusion/blob/a6fd5cc840d1b01ada8a48c7f2649789e86b256e/datafusion/datasource-parquet/src/access_plan.rs#L239
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
