[
https://issues.apache.org/jira/browse/SPARK-23802?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiao Li resolved SPARK-23802.
-----------------------------
Resolution: Fixed
Assignee: Robert Kruszewski
Fix Version/s: 2.3.1
> PropagateEmptyRelation can leave query plan in unresolved state
> ---------------------------------------------------------------
>
> Key: SPARK-23802
> URL: https://issues.apache.org/jira/browse/SPARK-23802
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 2.3.0
> Reporter: Robert Kruszewski
> Assignee: Robert Kruszewski
> Priority: Minor
> Fix For: 2.3.1
>
>
> Since [https://github.com/apache/spark/pull/19825] PropagateEmptyRelation has
> been taught to handle more cases it can cause the optimized query plan to be
> unresolved.
> Simple repro is to run following through the optimizer
> {code:java}
> LocalRelation.fromExternalRows(Seq('a.int), data = Seq(Row(1)))
> .join(LocalRelation('a.int, 'b.int), UsingJoin(FullOuter, "a" :: Nil),
> None){code}
> Which results in
> {code:java}
> Project [coalesce(a#0, null) AS a#7, null AS b#6]
> +- LocalRelation [a#0]{code}
> This then fails type check on coalesce expression since `a` and null have
> different type.
>
> Simple, targeted fix is to change PropagateEmptyRelation to add casts around
> nulls. More comprehensive fix would be to run type coercion at the end of
> optimization so it can fix cases like those. Alternatively the type checking
> code could treat NullType as equal to any other type and not fail the type
> check in the first place.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
