Repository: spark Updated Branches: refs/heads/branch-1.6 9a18115a8 -> 5cc1e2cec
[SPARK-10722] RDDBlockId not found in driver-heartbeater ## What changes were proposed in this pull request? To ensure that the deserialization of TaskMetrics uses a ClassLoader that knows about RDDBlockIds. The problem occurs only very rarely since it depends on which thread of the thread pool is used for the heartbeat. I observe that the code in question has been largely rewritten for v2.0.0 of Spark and the problem no longer manifests. However it would seem reasonable to fix this for those users who need to continue with the 1.6 version for some time yet. Hence I have created a fix for the 1.6 code branch. ## How was this patch tested? Due to the nature of the problem a reproducible testcase is difficult to produce. This problem was causing our application's nightly integration tests to fail randomly. Since applying the fix the tests have not failed due to this problem, for nearly six weeks now. Author: Simon Scott <simon.sc...@viavisolutions.com> Closes #13222 from simonjscott/fix-10722. Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/5cc1e2ce Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/5cc1e2ce Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/5cc1e2ce Branch: refs/heads/branch-1.6 Commit: 5cc1e2cec71ef18c76973608e909b4f37fcfcf6b Parents: 9a18115 Author: Simon Scott <simon.sc...@viavisolutions.com> Authored: Thu May 26 08:13:28 2016 -0500 Committer: Sean Owen <so...@cloudera.com> Committed: Thu May 26 08:13:28 2016 -0500 ---------------------------------------------------------------------- core/src/main/scala/org/apache/spark/executor/Executor.scala | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/spark/blob/5cc1e2ce/core/src/main/scala/org/apache/spark/executor/Executor.scala ---------------------------------------------------------------------- diff --git a/core/src/main/scala/org/apache/spark/executor/Executor.scala b/core/src/main/scala/org/apache/spark/executor/Executor.scala index a3ebaff..1131f8d 100644 --- a/core/src/main/scala/org/apache/spark/executor/Executor.scala +++ b/core/src/main/scala/org/apache/spark/executor/Executor.scala @@ -459,7 +459,9 @@ private[spark] class Executor( // JobProgressListener will hold an reference of it during // onExecutorMetricsUpdate(), then JobProgressListener can not see // the changes of metrics any more, so make a deep copy of it - val copiedMetrics = Utils.deserialize[TaskMetrics](Utils.serialize(metrics)) + val copiedMetrics = Utils.deserialize[TaskMetrics]( + Utils.serialize(metrics), + Utils.getContextOrSparkClassLoader) tasksMetrics += ((taskRunner.taskId, copiedMetrics)) } else { // It will be copied by serialization --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
