Repository: spark Updated Branches: refs/heads/master 3647120a5 -> fde1340c7
[SPARK-14701][STREAMING] First stop the event loop, then stop the checkpoint writer in JobGenerator Currently if we call `streamingContext.stop` (e.g. in a `StreamingListener.onBatchCompleted` callback) when a batch is about to complete, a `rejectedException` may get thrown from `checkPointWriter.executor`, since the `eventLoop` will try to process `DoCheckpoint` events even after the `checkPointWriter.executor` was stopped. Please see [SPARK-14701](https://issues.apache.org/jira/browse/SPARK-14701) for details and stack traces. ## What changes were proposed in this pull request? Reversed the stopping order of `event loop` and `checkpoint writer`. ## How was this patch tested? Existing test suits. (no dedicated test suits were added because the change is simple to reason about) Author: Liwei Lin <lwl...@gmail.com> Closes #12489 from lw-lin/spark-14701. Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/fde1340c Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/fde1340c Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/fde1340c Branch: refs/heads/master Commit: fde1340c768e18e9628e5f0eeb2f283c74c294fa Parents: 3647120 Author: Liwei Lin <lwl...@gmail.com> Authored: Fri Apr 22 15:42:47 2016 -0700 Committer: Shixiong Zhu <shixi...@databricks.com> Committed: Fri Apr 22 15:42:47 2016 -0700 ---------------------------------------------------------------------- .../org/apache/spark/streaming/scheduler/JobGenerator.scala | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/spark/blob/fde1340c/streaming/src/main/scala/org/apache/spark/streaming/scheduler/JobGenerator.scala ---------------------------------------------------------------------- diff --git a/streaming/src/main/scala/org/apache/spark/streaming/scheduler/JobGenerator.scala b/streaming/src/main/scala/org/apache/spark/streaming/scheduler/JobGenerator.scala index 307ff1f..8f9421f 100644 --- a/streaming/src/main/scala/org/apache/spark/streaming/scheduler/JobGenerator.scala +++ b/streaming/src/main/scala/org/apache/spark/streaming/scheduler/JobGenerator.scala @@ -154,9 +154,9 @@ class JobGenerator(jobScheduler: JobScheduler) extends Logging { graph.stop() } - // Stop the event loop and checkpoint writer - if (shouldCheckpoint) checkpointWriter.stop() + // First stop the event loop, then stop the checkpoint writer; see SPARK-14701 eventLoop.stop() + if (shouldCheckpoint) checkpointWriter.stop() logInfo("Stopped JobGenerator") } --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
