Github user JoshRosen commented on the pull request:
https://github.com/apache/spark/pull/3638#issuecomment-68978135
Alright, this looks good to me and I'd like to merge it. I'll revise the
commit message to more accurately describe the actual change that's being
committed. I'm thinking of something like this (incorporating pieces from the
JIRA):
> Currently, Spark assumes that serialization cannot fail when tasks are
serialized in the TaskSetManager. We assume this because upstream, in the
DAGScheduler, we attempt to catch any serialization errors by testing whether
the first task / partition can be serialized. However, in some cases this
upstream test is not sufficient - i.e. an RDD's first partition might be
serializable even though other partitions are not.
> This patch solves this problem by catching serialization errors at the
time that TaskSetManager attempts to launch tasks. If a task fails with a
serialization error, TaskSetManager will now abort task's task set. This
prevents uncaught serialization errors from crashing the DAGScheduler.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
