stuhood commented on code in PR #25279:
URL: https://github.com/apache/datafusion/pull/25279#discussion_r4067269658
##########
datafusion/datasource/src/file_scan_config/mod.rs:
##########
@@ -617,54 +615,15 @@ impl From<FileScanConfig> for FileScanConfigBuilder {
}
}
-/// Builds `Partitioning::Hash` over `partition_cols` (resolved to their
indices in
-/// `schema`). Returns `None` when there are no partition columns.
-///
-/// # Deprecated
-/// Use [`range_partitioning_from_partition_fields`] instead.
-#[deprecated(
- since = "56.0.0",
- note = "Hive file groups are value partitioned, not hash partitioned. Use
range_partitioning_from_partition_fields"
-)]
+/// Builds output partitioning over `partition_cols` (resolved to their
indices in
+/// `schema`) with `partition_count` partitions. Returns `None` when there are
no
+/// partition columns. Callers use this to declare the output partitioning of
a scan
+/// whose file groups are organized by partition column values.
pub fn output_partitioning_from_partition_fields(
schema: &Schema,
partition_cols: &Fields,
partition_count: usize,
) -> Option<Partitioning> {
Review Comment:
This still returns `Partioning::Hash` by default, which means external
consumers might still get the old behavior.
Should it be deprecated and replaced with a new function/method like?:
```rust
pub fn output_range_partitioning_from_split_points(
schema: &Schema,
partition_cols: &Fields,
split_points: Vec<SplitPoint>,
) -> Result<Option<Partitioning>>;
```
...which could then be used in `ListingTable::scan` as well?
##########
datafusion/catalog-listing/src/table.rs:
##########
@@ -683,8 +683,26 @@ impl ListingTable {
None => {} // no ordering required
}
+ // Hive grouped files are contiguous key intervals, so they declare
`Range`
+ // unless the statistics re-cut above changed the groups. The ordering
must
+ // match the `SortOptions::default()` used to cut the groups.
+ let derived_output_partitioning =
+ if partitioned_by_file_group && !regrouped_by_statistics {
+ let ordering = table_partition_cols
+ .iter()
+ .map(|field| {
+
Expr::Column(Column::from_name(field.name())).sort(true, true)
Review Comment:
If these bools are meant to correspond to `SortOptions::default()`, maybe
they should be derived from it instead.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
