[jira] [Commented] (SPARK-14846) Driver process fails to terminate when graceful shutdown is used
[ https://issues.apache.org/jira/browse/SPARK-14846?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15255375#comment-15255375 ] Mattias Aspholm commented on SPARK-14846: - Yes, you're right of course. Sorry about that. I'm still having problems with the driver not closing down in graceful (even though there's no work left), but I realise now my initial conclusions was bad, the reason why it hangs in awaitTermination is that the termination condition is not signaled. I need to find out why that happens. Ok for me to close this bug as invalid. I'll file another one if it turns out to be some bug after all. > Driver process fails to terminate when graceful shutdown is used > > > Key: SPARK-14846 > URL: https://issues.apache.org/jira/browse/SPARK-14846 > Project: Spark > Issue Type: Bug > Components: Streaming >Affects Versions: 1.6.1 >Reporter: Mattias Aspholm > > During shutdown, the job scheduler in Streaming (JobScheduler.stop) spends > some time waiting for all queued work to complete. If graceful shutdown is > used, the time is 1 hour, for non-graceful shutdown it's 2 seconds. > The wait is implemented using the ThreadPoolExecutor.awaitTermination method > in java.util.concurrent. The problem is that instead of looping over the > method for the desired period of time, the wait period is passed in as the > timeout parameter to awaitTermination. > The result is that if the termination condition is false the first time, the > method will sleep for the timeout period before trying again. In the case of > graceful shutdown this means at least an hour's wait before the condition is > checked again, even though all work is completed in just a few seconds. The > driver process will continue to live during this time. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-14846) Driver process fails to terminate when graceful shutdown is used
[ https://issues.apache.org/jira/browse/SPARK-14846?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15255223#comment-15255223 ] Sean Owen commented on SPARK-14846: --- No, that's not what methods like awaitNanos do in the JDK classes. It waits for up to that time, but the normal mechanism is that the Condition is signaled before the timeout occurs. This is not a sleep-and-poll. > Driver process fails to terminate when graceful shutdown is used > > > Key: SPARK-14846 > URL: https://issues.apache.org/jira/browse/SPARK-14846 > Project: Spark > Issue Type: Bug > Components: Streaming >Affects Versions: 1.6.1 >Reporter: Mattias Aspholm > > During shutdown, the job scheduler in Streaming (JobScheduler.stop) spends > some time waiting for all queued work to complete. If graceful shutdown is > used, the time is 1 hour, for non-graceful shutdown it's 2 seconds. > The wait is implemented using the ThreadPoolExecutor.awaitTermination method > in java.util.concurrent. The problem is that instead of looping over the > method for the desired period of time, the wait period is passed in as the > timeout parameter to awaitTermination. > The result is that if the termination condition is false the first time, the > method will sleep for the timeout period before trying again. In the case of > graceful shutdown this means at least an hour's wait before the condition is > checked again, even though all work is completed in just a few seconds. The > driver process will continue to live during this time. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-14846) Driver process fails to terminate when graceful shutdown is used
[ https://issues.apache.org/jira/browse/SPARK-14846?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15254628#comment-15254628 ] Shixiong Zhu commented on SPARK-14846: -- `awaitTermination` doesn't need to wait at least one hour. `jobExecutor.shutdown()` is called before `jobExecutor.awaitTermination`. So when all threads in `jobExecutor` are done, awaitTermination will return. > Driver process fails to terminate when graceful shutdown is used > > > Key: SPARK-14846 > URL: https://issues.apache.org/jira/browse/SPARK-14846 > Project: Spark > Issue Type: Bug > Components: Streaming >Affects Versions: 1.6.1 >Reporter: Mattias Aspholm > > During shutdown, the job scheduler in Streaming (JobScheduler.stop) spends > some time waiting for all queued work to complete. If graceful shutdown is > used, the time is 1 hour, for non-graceful shutdown it's 2 seconds. > The wait is implemented using the ThreadPoolExecutor.awaitTermination method > in java.util.concurrent. The problem is that instead of looping over the > method for the desired period of time, the wait period is passed in as the > timeout parameter to awaitTermination. > The result is that if the termination condition is false the first time, the > method will sleep for the timeout period before trying again. In the case of > graceful shutdown this means at least an hour's wait before the condition is > checked again, even though all work is completed in just a few seconds. The > driver process will continue to live during this time. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
