[
https://issues.apache.org/jira/browse/HIVE-18831?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16416004#comment-16416004
]
Sahil Takiar commented on HIVE-18831:
-------------------------------------
Hey [~lirui], I agree a {{Throwable}} is the most natural thing to expect when
something goes wrong, and ideally the exact {{Throwable}} thrown by the Spark
job would be propagated to the HS2 Job Monitor.
I tried to implement what SPARK-8625 was doing. Attempt to serialize the
exception, and fallback to old behavior if that fails. However, that seemed to
be quite tricky given how the RPC layer works. I would have to make some
invasive changes to {{Rpc#call}} as right now it closes the connection whenever
it fails to send a given object. I didn't think making all the necessary
changes to the {{Rpc}} class was worth it, which is why I went down the
"wrapper" approach. And I believe there are some exceptions that will fail to
get serialized, e.g. Scala's {{MatchError}} includes the objects that couldn't
get matched.
One middle ground would be to make {{SparkJobExceptionWrapper}} extend
{{Exception}}, and the {{getMessage()}} call would return the {{rootCause}}.
The class would then mainly be present to wrap the String representation of the
full stack-trace (accessible via {{getFullStackTrace()}}).
> Differentiate errors that are thrown by Spark tasks
> ---------------------------------------------------
>
> Key: HIVE-18831
> URL: https://issues.apache.org/jira/browse/HIVE-18831
> Project: Hive
> Issue Type: Sub-task
> Components: Spark
> Reporter: Sahil Takiar
> Assignee: Sahil Takiar
> Priority: Major
> Attachments: HIVE-18831.1.patch, HIVE-18831.2.patch,
> HIVE-18831.3.patch, HIVE-18831.4.patch, HIVE-18831.6.patch, HIVE-18831.7.patch
>
>
> We propagate exceptions from Spark task failures to the client well, but we
> don't differentiate between errors from HS2 / RSC vs. errors thrown by
> individual tasks.
> Main motivation is that when the client sees a propagated Spark exception its
> difficult to know what part of the excution threw the exception.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
