[
https://issues.apache.org/jira/browse/SPARK-59981?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Toth updated SPARK-59981:
-------------------------------
Affects Version/s: 4.4.0
(was: 4.2.0)
Description:
A V2 scan that reports a {{KeyedPartitioning}} gets an ordering over its
partition key expressions (SPARK-56241), since all rows of a partition have the
same key. It gets it when it reports no ordering, or when Spark ignores its
report (SPARK-59721). It does not get it when it reports an empty ordering, or
one Spark keeps nothing of:
* a report that starts with a column pruned from the scan output (SPARK-59899)
and has no later sort order on a partition key other than a transform;
* a report that starts with a partition transform (SPARK-59995) and has no
later sort order on a partition key other than a transform.
The ordering over the partition keys follows from the partitioning, not from
the report. So it can be derived in these cases too. A sort-merge join, a
window or a sort aggregate on the partition keys then needs no sort above the
scan.
was:
With {{spark.sql.sources.v2.bucketing.partitionKeyOrdering.enabled}} on (the
default since SPARK-59396), a V2 scan that reports a {{KeyGroupedPartitioning}}
gets an output ordering over its partition key expressions (SPARK-56241). That
works when the scan does not implement {{SupportsReportOrdering}}, or when
Spark ignores the ordering it reports (SPARK-59721). It does not work when the
scan implements {{SupportsReportOrdering}} and reports an empty ordering:
* {{V2ScanPartitioningAndOrdering}} stores the empty report as {{Some(Nil)}}.
* {{DataSourceV2ScanExecBase.outputOrdering}} takes its {{case (Some(o), p)}}
branch before the derivation, so the scan reports no ordering at all.
Measured with a scan that reports {{identity(id)}}, joined on {{id}} with
another {{identity(id)}} table: the sort-merge join keeps a sort above the
scan. When the same scan reports an ordering that Spark ignores, the ordering
is derived and the sort goes away. The results are correct either way.
The conf's doc covers this case: "a V2 data source that reports a
KeyedPartitioning but does not report explicit ordering via
SupportsReportOrdering".
Found in the review of https://github.com/apache/spark/pull/58979.
The same happens when the report keeps no sort order:
* a report that starts with a sort order on a column pruned from the scan
output and has no sort order on a partition key (SPARK-59899 restricts the
report to the output);
* a report that starts with a sort order over a partition transform and has no
sort order on a partition key that is not a transform (SPARK-59995 drops those).
Issue Type: Improvement (was: Bug)
Summary: Derive the partition-key ordering when a V2 scan's
reported ordering keeps no sort order (was: A V2 scan whose reported ordering
keeps no sort order loses the ordering derived from its partition keys)
> Derive the partition-key ordering when a V2 scan's reported ordering keeps no
> sort order
> ----------------------------------------------------------------------------------------
>
> Key: SPARK-59981
> URL: https://issues.apache.org/jira/browse/SPARK-59981
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 4.4.0
> Reporter: Peter Toth
> Priority: Major
> Labels: pull-request-available
>
> A V2 scan that reports a {{KeyedPartitioning}} gets an ordering over its
> partition key expressions (SPARK-56241), since all rows of a partition have
> the same key. It gets it when it reports no ordering, or when Spark ignores
> its report (SPARK-59721). It does not get it when it reports an empty
> ordering, or one Spark keeps nothing of:
> * a report that starts with a column pruned from the scan output
> (SPARK-59899) and has no later sort order on a partition key other than a
> transform;
> * a report that starts with a partition transform (SPARK-59995) and has no
> later sort order on a partition key other than a transform.
> The ordering over the partition keys follows from the partitioning, not from
> the report. So it can be derived in these cases too. A sort-merge join, a
> window or a sort aggregate on the partition keys then needs no sort above the
> scan.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
