[
https://issues.apache.org/jira/browse/SPARK-14846?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15254628#comment-15254628
]
Shixiong Zhu commented on SPARK-14846:
--------------------------------------
`awaitTermination` doesn't need to wait at least one hour.
`jobExecutor.shutdown()` is called before `jobExecutor.awaitTermination`. So
when all threads in `jobExecutor` are done, awaitTermination will return.
> Driver process fails to terminate when graceful shutdown is used
> ----------------------------------------------------------------
>
> Key: SPARK-14846
> URL: https://issues.apache.org/jira/browse/SPARK-14846
> Project: Spark
> Issue Type: Bug
> Components: Streaming
> Affects Versions: 1.6.1
> Reporter: Mattias Aspholm
>
> During shutdown, the job scheduler in Streaming (JobScheduler.stop) spends
> some time waiting for all queued work to complete. If graceful shutdown is
> used, the time is 1 hour, for non-graceful shutdown it's 2 seconds.
> The wait is implemented using the ThreadPoolExecutor.awaitTermination method
> in java.util.concurrent. The problem is that instead of looping over the
> method for the desired period of time, the wait period is passed in as the
> timeout parameter to awaitTermination.
> The result is that if the termination condition is false the first time, the
> method will sleep for the timeout period before trying again. In the case of
> graceful shutdown this means at least an hour's wait before the condition is
> checked again, even though all work is completed in just a few seconds. The
> driver process will continue to live during this time.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
