[
https://issues.apache.org/jira/browse/SPARK-31078?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wenchen Fan reassigned SPARK-31078:
-----------------------------------
Assignee: Terry Kim
> outputOrdering should handle aliases correctly
> ----------------------------------------------
>
> Key: SPARK-31078
> URL: https://issues.apache.org/jira/browse/SPARK-31078
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 3.0.0
> Reporter: Terry Kim
> Assignee: Terry Kim
> Priority: Major
>
> Currently, `outputOrdering` doesn't respect aliases. Thus, the following
> would produce an unnecessary sort node:
> {code:java}
> withSQLConf(SQLConf.AUTO_BROADCASTJOIN_THRESHOLD.key -> "0") {
> val df = (0 until 20).toDF("i").as("df")
> df.repartition(8, df("i")).write.format("parquet")
> .bucketBy(8, "i").sortBy("i").saveAsTable("t")
> val t1 = spark.table("t")
> val t2 = t1.selectExpr("i as ii")
> t1.join(t2, t1("i") === t2("ii")).explain
> }
> {code}
> would produce an unnecessary sort node:
> {code:java}
> == Physical Plan ==
> *(3) SortMergeJoin [i#8], [ii#10], Inner
> :- *(1) Project [i#8]
> : +- *(1) Filter isnotnull(i#8)
> : +- *(1) ColumnarToRow
> : +- FileScan parquet default.t[i#8] Batched: true, DataFilters:
> [isnotnull(i#8)], Format: Parquet, Location: InMemoryFileIndex[file:/...,
> PartitionFilters: [], PushedFilters: [IsNotNull(i)], ReadSchema:
> struct<i:int>, SelectedBucketsCount: 8 out of 8
> +- *(2) Sort [ii#10 ASC NULLS FIRST], false, 0
> +- *(2) Project [i#8 AS ii#10]
> +- *(2) Filter isnotnull(i#8)
> +- *(2) ColumnarToRow
> +- FileScan parquet default.t[i#8] Batched: true, DataFilters:
> [isnotnull(i#8)], Format: Parquet, Location: InMemoryFileIndex[file:/...,
> PartitionFilters: [], PushedFilters: [IsNotNull(i)], ReadSchema:
> struct<i:int>, SelectedBucketsCount: 8 out of 8
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
