[
https://issues.apache.org/jira/browse/SPARK-17271?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Herman van Hovell resolved SPARK-17271.
---------------------------------------
Resolution: Fixed
Assignee: Tejas Patil
Fix Version/s: 2.1.0
> Planner adds un-necessary Sort even if child ordering is semantically same as
> required ordering
> -----------------------------------------------------------------------------------------------
>
> Key: SPARK-17271
> URL: https://issues.apache.org/jira/browse/SPARK-17271
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 1.6.2, 2.0.0
> Reporter: Tejas Patil
> Assignee: Tejas Patil
> Fix For: 2.1.0
>
>
> Found a case when the planner is adding un-needed SORT operation due to bug
> in the way comparison for `SortOrder` is done at
> https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/exchange/EnsureRequirements.scala#L253
> `SortOrder` needs to be compared semantically because `Expression` within two
> `SortOrder` can be "semantically equal" but not literally equal objects.
> eg. In case of `sql("SELECT * FROM table1 a JOIN table2 b ON a.col1=b.col1")`
> Expression in required SortOrder:
> {code}
> AttributeReference(
> name = "col1",
> dataType = LongType,
> nullable = false
> ) (exprId = exprId,
> qualifier = Some("a")
> )
> {code}
> Expression in child SortOrder:
> {code}
> AttributeReference(
> name = "col1",
> dataType = LongType,
> nullable = false
> ) (exprId = exprId)
> {code}
> Notice that the output column has a qualifier but the child attribute does
> not but the inherent expression is the same and hence in this case we can say
> that the child satisfies the required sort order.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
