adriangb commented on code in PR #20043:
URL: https://github.com/apache/datafusion/pull/20043#discussion_r2742158292
##########
datafusion/datasource/src/file.rs:
##########
@@ -189,7 +189,21 @@ pub trait FileSource: Send + Sync {
/// * `Inexact` - Created a source optimized for ordering (e.g., reversed
row groups) but not perfectly sorted
/// * `Unsupported` - Cannot optimize for this ordering
///
- /// Default implementation returns `Unsupported`.
+ /// Default implementation delegates to [`Self::try_reverse_output`].
Review Comment:
Maybe state the version / timeline at which we will remove
`try_reverse_output` and what the default impl of `try_pushdown_sort` will be
at that point.
Also we should make an explicit recommendation for users: should they
immediately implement `try_pushdown_sort`?
##########
datafusion/datasource/src/file_scan_config.rs:
##########
@@ -1157,19 +1157,30 @@ impl FileScanConfig {
&self,
new_file_source: Arc<dyn FileSource>,
is_exact: bool,
+ order: &[PhysicalSortExpr],
) -> Result<Arc<dyn DataSource>> {
let mut new_config = self.clone();
- // Reverse file groups (FileScanConfig's responsibility)
- new_config.file_groups = new_config
- .file_groups
- .into_iter()
- .map(|group| {
- let mut files = group.into_inner();
- files.reverse();
- files.into()
- })
- .collect();
+ // Reverse file groups (FileScanConfig's responsibility) if doing so
helps satisfy the
+ // requested ordering.
+ let reverse_file_groups =
+ LexOrdering::new(order.iter().cloned()).is_some_and(|requested| {
+ self.output_ordering
+ .iter()
+ .any(|ordering| ordering.is_reverse(&requested))
+ });
Review Comment:
Can you explain this bit? I can't tell where this code is being moved from.
Is it new / handling a bug?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
