[ 
https://issues.apache.org/jira/browse/SPARK-60043?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ASF GitHub Bot updated SPARK-60043:
-----------------------------------
    Labels: pull-request-available  (was: )

> Keep the sort orders after a partition-key transform in a V2 scan's reported 
> ordering
> -------------------------------------------------------------------------------------
>
>                 Key: SPARK-60043
>                 URL: https://issues.apache.org/jira/browse/SPARK-60043
>             Project: Spark
>          Issue Type: Improvement
>          Components: SQL
>    Affects Versions: 4.2.0
>            Reporter: Peter Toth
>            Priority: Major
>              Labels: pull-request-available
>
> SPARK-59995 drops every sort order that holds a partition transform from a V2 
> scan's output ordering. In a reported ordering, such a sort order also ends 
> the leading run of sort orders that the scan keeps. A sort order on a 
> transform that is itself a partition key is constant within each partition, 
> like any partition key, so the sort orders after it still hold. They are 
> dropped anyway:
> * keys {{[days(ts)]}}, reported {{[days(ts), id]}}: the scan reports {{[]}} 
> instead of {{[id]}};
> * keys {{[years(ts), id]}}, reported {{[years(ts), id, name]}}: {{[id]}} 
> instead of {{[id, name]}};
> * keys {{[days(ts), id]}}, reported {{[days(ts)]}}: {{[]}}, while no report 
> would derive {{[id]}}.
> Skipping such a sort order instead of ending the run would keep them. 
> Matching a transform sort order to a transform key needs {{isSameFunction}} 
> plus semantically equal children rather than {{ExpressionSet}}, since the two 
> are bound separately and {{BoundFunction.equals}} is only a SHOULD. Falling 
> back to the derived ordering when nothing of a non-empty report survives 
> would cover the last case.
> This is not a regression, since none of these orderings satisfied a 
> requirement before SPARK-59995. Measured on the SPARK-59995 PR: two tables 
> that report {{[years(ts), ts]}} over the key {{years(ts)}}, joined on {{ts}} 
> with {{preserveOrderingOnCoalesce}} on, get 2 {{SortExec}}s. Skipping the 
> key's sort order removes both, with correct rows.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to