crepererum opened a new issue, #17354: URL: https://github.com/apache/datafusion/issues/17354
### Describe the bug Currently the output ordering -- which is defined per partition -- is a vector of `LexOrdering`: https://github.com/apache/datafusion/blob/d19bf524e384bc24e509c70f1806b6f330829529/datafusion/datasource/src/file_scan_config.rs#L183-L184 This was kinda OK before #16217 because `LexOrdering` was allowed to be empty -- which was basically a sentinel value for "not ordering". Now however the code cannot specify that a partition isn't ordered anymore. This actually leads to some funky bugs like in this code here: https://github.com/apache/datafusion/blob/d19bf524e384bc24e509c70f1806b6f330829529/datafusion/datasource/src/file_scan_config.rs#L1382-L1384 So if the projection for a single partition leads to "unordered", then `get_projected_output_ordering(input).len() < input.len()`, i.e. we loose partitions. This can clearly not be right. ### To Reproduce _No response_ ### Expected behavior `FileScanConfig::output_ordering` must be `Vec<Option<LexOrdering>>`. ### Additional context _No response_ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
