[
https://issues.apache.org/jira/browse/SPARK-14846?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15255375#comment-15255375
]
Mattias Aspholm edited comment on SPARK-14846 at 4/23/16 8:23 PM:
------------------------------------------------------------------
You're right of course. Sorry about that. I'm still having problems with the
driver not closing down in graceful (even though there's no work left), but I
realise now my initial conclusions was bad, the reason why it hangs in
awaitTermination is that the termination condition is not signaled. I need to
find out why that happens.
Ok for me to close this bug as invalid. I'll file another one if it turns out
to be some bug after all.
was (Author: masph...@gmail.com):
Yes, you're right of course. Sorry about that. I'm still having problems with
the driver not closing down in graceful (even though there's no work left), but
I realise now my initial conclusions was bad, the reason why it hangs in
awaitTermination is that the termination condition is not signaled. I need to
find out why that happens.
Ok for me to close this bug as invalid. I'll file another one if it turns out
to be some bug after all.
> Driver process fails to terminate when graceful shutdown is used
> ----------------------------------------------------------------
>
> Key: SPARK-14846
> URL: https://issues.apache.org/jira/browse/SPARK-14846
> Project: Spark
> Issue Type: Bug
> Components: Streaming
> Affects Versions: 1.6.1
> Reporter: Mattias Aspholm
>
> During shutdown, the job scheduler in Streaming (JobScheduler.stop) spends
> some time waiting for all queued work to complete. If graceful shutdown is
> used, the time is 1 hour, for non-graceful shutdown it's 2 seconds.
> The wait is implemented using the ThreadPoolExecutor.awaitTermination method
> in java.util.concurrent. The problem is that instead of looping over the
> method for the desired period of time, the wait period is passed in as the
> timeout parameter to awaitTermination.
> The result is that if the termination condition is false the first time, the
> method will sleep for the timeout period before trying again. In the case of
> graceful shutdown this means at least an hour's wait before the condition is
> checked again, even though all work is completed in just a few seconds. The
> driver process will continue to live during this time.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
