[
https://issues.apache.org/jira/browse/SPARK-20872?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kris Mok updated SPARK-20872:
-----------------------------
Description:
A ShuffleExchange's coordinator can be null sometimes, and when we need to do a
toString() on it, it'll go to ShuffleExchange.nodeName() and throw a MatchError
there because of inexhaustive match -- the match only handles Some and None,
but doesn't handle null.
An example of this issue is when trying to inspect a Catalyst physical operator
on the Executor side in an IDE:
{code:none}
child = {WholeStageCodegenExec@13881} Method threw 'scala.MatchError'
exception. Cannot evaluate
org.apache.spark.sql.execution.WholeStageCodegenExec.toString()
{code}
where this WholeStageCodegenExec transitively references a ShuffleExchange.
On the Executor side, this ShuffleExchange instance is deserialized from the
data sent over from the Driver, and because the coordinator field is marked
transient, it's not carried over to the Executor, that's why it can be null
upon inspection.
was:
A ShuffleExchange's coordinator can be null sometimes, and when we need to do a
toString() on it, it'll go to ShuffleExchange.nodeName() and throw a MatchError
there because of inexhaustive match -- the match only handles Some and None,
but doesn't handle null.
An example of this issue is when trying to inspect a Catalyst physical operator
in an IDE:
{code:none}
child = {WholeStageCodegenExec@13881} Method threw 'scala.MatchError'
exception. Cannot evaluate
org.apache.spark.sql.execution.WholeStageCodegenExec.toString()
{code}
where this WholeStageCodegenExec transitively references a ShuffleExchange.
> ShuffleExchange.nodeName should handle null coordinator
> -------------------------------------------------------
>
> Key: SPARK-20872
> URL: https://issues.apache.org/jira/browse/SPARK-20872
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 2.2.0
> Reporter: Kris Mok
> Priority: Trivial
> Labels: easyfix
>
> A ShuffleExchange's coordinator can be null sometimes, and when we need to do
> a toString() on it, it'll go to ShuffleExchange.nodeName() and throw a
> MatchError there because of inexhaustive match -- the match only handles Some
> and None, but doesn't handle null.
> An example of this issue is when trying to inspect a Catalyst physical
> operator on the Executor side in an IDE:
> {code:none}
> child = {WholeStageCodegenExec@13881} Method threw 'scala.MatchError'
> exception. Cannot evaluate
> org.apache.spark.sql.execution.WholeStageCodegenExec.toString()
> {code}
> where this WholeStageCodegenExec transitively references a ShuffleExchange.
> On the Executor side, this ShuffleExchange instance is deserialized from the
> data sent over from the Driver, and because the coordinator field is marked
> transient, it's not carried over to the Executor, that's why it can be null
> upon inspection.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
