[ https://issues.apache.org/jira/browse/SPARK-45099?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Huw updated SPARK-45099: ------------------------ Description: When performing a 'using' join with a sort hint in a full outer, the ResolveNaturalAndUsingJoin will kick in and build a new join with Equality conditions and a Projection like this: {quote}val joinedCols = joinPairs.map \{ case (l, r) => Alias(Coalesce(Seq(l, r)), l.name)() } {quote} There's nothing wrong with this per se, but, SortMergeJoinExec has it's output ordering for a full outer join as empty, even though these join pairs in their final coalesced form actually are ordered. This means that code like this: {quote}{{frames.reduceLeft }}{{{}{ case (l, r) => l.join(r.hint("merge"), usingColumns = Seq("a", "b"), joinType = "outer") }{}}}{{{}{}}}{quote} Given a non empty list of frames, will not 'stream' without a shuffle step, as each join forgets its sort order. Ideally this whole operation wouldn't require any shuffles if all the frames are grouped and sorted by the keys. was: When performing a 'using' join with a sort hint in a full outer, the ResolveNaturalAndUsingJoin will kick in and build a new join with Equality conditions and a Projection like this: {quote}val joinedCols = joinPairs.map \{ case (l, r) => Alias(Coalesce(Seq(l, r)), l.name)() } {quote} There's nothing wrong with this per se, but, SortMergeJoinExec has it's output ordering for a full outer join as empty, even though these join pairs in their final coalesced form actually are ordered. This means that code like this: {quote}frames.reduceLeft({ case (l, r) => l.join(r.hint("merge"), usingColumns = Seq("a", "b"), joinType = "outer") }) {quote} Given a non empty list of frames, will not 'stream' without a shuffle step, as each join forgets its sort order. Ideally this whole operation wouldn't require any shuffles if all the frames are grouped and sorted by the keys. > SortMergeExec with Outer using join forgets sort information > ------------------------------------------------------------ > > Key: SPARK-45099 > URL: https://issues.apache.org/jira/browse/SPARK-45099 > Project: Spark > Issue Type: Improvement > Components: SQL > Affects Versions: 3.4.1 > Reporter: Huw > Priority: Minor > > When performing a 'using' join with a sort hint in a full outer, the > ResolveNaturalAndUsingJoin will kick in and build a new join with Equality > conditions and a Projection like this: > {quote}val joinedCols = joinPairs.map \{ case (l, r) => Alias(Coalesce(Seq(l, > r)), l.name)() } > {quote} > There's nothing wrong with this per se, but, SortMergeJoinExec has it's > output ordering for a full outer join as empty, even though these join pairs > in their final coalesced form actually are ordered. > This means that code like this: > {quote}{{frames.reduceLeft }}{{{}{ > case (l, r) => l.join(r.hint("merge"), usingColumns = Seq("a", "b"), > joinType = "outer") > }{}}}{{{}{}}}{quote} > Given a non empty list of frames, will not 'stream' without a shuffle step, > as each join forgets its sort order. > Ideally this whole operation wouldn't require any shuffles if all the frames > are grouped and sorted by the keys. -- This message was sent by Atlassian Jira (v8.20.10#820010) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org