alamb commented on code in PR #4170:
URL: https://github.com/apache/arrow-datafusion/pull/4170#discussion_r1020748593
##########
datafusion/core/src/datasource/listing/table.rs:
##########
@@ -220,6 +225,16 @@ pub struct ListingOptions {
/// Group files to avoid that the number of partitions exceeds
/// this limit
pub target_partitions: usize,
+ /// Optional pre-known sort order. Must be `SortExpr`s.
+ ///
+ /// DataFusion may take advantage of this ordering to omit sorts
+ /// or use more efficient algorithms. Currently sortedness must be
+ /// provided if it is known by some external mechanism, but may in
+ /// the future be automatically determined, for example using
+ /// parquet metadata.
+ ///
+ /// See <https://github.com/apache/arrow-datafusion/issues/4177>
+ pub file_sort_order: Option<Vec<Expr>>,
Review Comment:
🤔 after thinking about this over the evening I think I like option 1
(ignore provided sort order if there is more than one file in each partition)
the best
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscr...@arrow.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
