[
https://issues.apache.org/jira/browse/SPARK-14846?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15255375#comment-15255375
]
Mattias Aspholm commented on SPARK-14846:
-----------------------------------------
Yes, you're right of course. Sorry about that. I'm still having problems with
the driver not closing down in graceful (even though there's no work left), but
I realise now my initial conclusions was bad, the reason why it hangs in
awaitTermination is that the termination condition is not signaled. I need to
find out why that happens.
Ok for me to close this bug as invalid. I'll file another one if it turns out
to be some bug after all.
> Driver process fails to terminate when graceful shutdown is used
> ----------------------------------------------------------------
>
> Key: SPARK-14846
> URL: https://issues.apache.org/jira/browse/SPARK-14846
> Project: Spark
> Issue Type: Bug
> Components: Streaming
> Affects Versions: 1.6.1
> Reporter: Mattias Aspholm
>
> During shutdown, the job scheduler in Streaming (JobScheduler.stop) spends
> some time waiting for all queued work to complete. If graceful shutdown is
> used, the time is 1 hour, for non-graceful shutdown it's 2 seconds.
> The wait is implemented using the ThreadPoolExecutor.awaitTermination method
> in java.util.concurrent. The problem is that instead of looping over the
> method for the desired period of time, the wait period is passed in as the
> timeout parameter to awaitTermination.
> The result is that if the termination condition is false the first time, the
> method will sleep for the timeout period before trying again. In the case of
> graceful shutdown this means at least an hour's wait before the condition is
> checked again, even though all work is completed in just a few seconds. The
> driver process will continue to live during this time.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
