[jira] [Comment Edited] (SPARK-10911) Executors should System.exit on clean shutdown
[
https://issues.apache.org/jira/browse/SPARK-10911?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15035752#comment-15035752
]
Jaromir Vanek edited comment on SPARK-10911 at 12/2/15 1:04 PM:
{quote}
why isn't YARN killing the executors?
{quote}
It's because in YARN there is also bug that prevents container from being
killed by process signal. Will be fixed in Hadoop 2.8.0
https://issues.apache.org/jira/browse/HADOOP-12441
was (Author: vanekjar):
{{quote}}
why isn't YARN killing the executors?
{{quote}}
It's because in YARN there is also bug that prevents container from being
killed by process signal. Will be fixed in Hadoop 2.8.0
https://issues.apache.org/jira/browse/HADOOP-12441
> Executors should System.exit on clean shutdown
> --
>
> Key: SPARK-10911
> URL: https://issues.apache.org/jira/browse/SPARK-10911
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core
>Affects Versions: 1.5.1
>Reporter: Thomas Graves
>Assignee: Zhuo Liu
>Priority: Minor
>
> Executors should call System.exit on clean shutdown to make sure all user
> threads exit and jvm shuts down.
> We ran into a case where an Executor was left around for days trying to
> shutdown because the user code was using a non-daemon thread pool and one of
> those threads wasn't exiting. We should force the jvm to go away with
> System.exit.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-10911) Executors should System.exit on clean shutdown
[
https://issues.apache.org/jira/browse/SPARK-10911?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15035752#comment-15035752
]
Jaromir Vanek commented on SPARK-10911:
---
{{quote}}
why isn't YARN killing the executors?
{{quote}}
It's because in YARN there is also bug that prevents container from being
killed by process signal. Will be fixed in Hadoop 2.8.0
https://issues.apache.org/jira/browse/HADOOP-12441
> Executors should System.exit on clean shutdown
> --
>
> Key: SPARK-10911
> URL: https://issues.apache.org/jira/browse/SPARK-10911
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core
>Affects Versions: 1.5.1
>Reporter: Thomas Graves
>Assignee: Zhuo Liu
>Priority: Minor
>
> Executors should call System.exit on clean shutdown to make sure all user
> threads exit and jvm shuts down.
> We ran into a case where an Executor was left around for days trying to
> shutdown because the user code was using a non-daemon thread pool and one of
> those threads wasn't exiting. We should force the jvm to go away with
> System.exit.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-10911) Executors should System.exit on clean shutdown
[
https://issues.apache.org/jira/browse/SPARK-10911?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14993565#comment-14993565
]
Jaromir Vanek commented on SPARK-10911:
---
I also came across this problem. Is there anything that would spark against
explicit calling {{System.exit(0)}} in the
{{CoarseGrainedExecutorBackend#main}} method?
> Executors should System.exit on clean shutdown
> --
>
> Key: SPARK-10911
> URL: https://issues.apache.org/jira/browse/SPARK-10911
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
>Affects Versions: 1.5.1
>Reporter: Thomas Graves
>
> Executors should call System.exit on clean shutdown to make sure all user
> threads exit and jvm shuts down.
> We ran into a case where an Executor was left around for days trying to
> shutdown because the user code was using a non-daemon thread pool and one of
> those threads wasn't exiting. We should force the jvm to go away with
> System.exit.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Comment Edited] (SPARK-10911) Executors should System.exit on clean shutdown
[
https://issues.apache.org/jira/browse/SPARK-10911?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14993565#comment-14993565
]
Jaromir Vanek edited comment on SPARK-10911 at 11/6/15 12:15 PM:
-
I also came across this problem. Is there anything that would speak against
explicit calling {{System.exit(0)}} in the
{{CoarseGrainedExecutorBackend#main}} method?
was (Author: vanekjar):
I also came across this problem. Is there anything that would spark against
explicit calling {{System.exit(0)}} in the
{{CoarseGrainedExecutorBackend#main}} method?
> Executors should System.exit on clean shutdown
> --
>
> Key: SPARK-10911
> URL: https://issues.apache.org/jira/browse/SPARK-10911
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
>Affects Versions: 1.5.1
>Reporter: Thomas Graves
>
> Executors should call System.exit on clean shutdown to make sure all user
> threads exit and jvm shuts down.
> We ran into a case where an Executor was left around for days trying to
> shutdown because the user code was using a non-daemon thread pool and one of
> those threads wasn't exiting. We should force the jvm to go away with
> System.exit.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Comment Edited] (SPARK-8393) JavaStreamingContext#awaitTermination() throws non-declared InterruptedException
[
https://issues.apache.org/jira/browse/SPARK-8393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14599185#comment-14599185
]
Jaromir Vanek edited comment on SPARK-8393 at 6/24/15 9:48 AM:
---
I think the suggested workaround is fine for the current 1.x version of Spark.
So updating the documentation would be proper solution to prevent other
developers from unexpected problems.
But in the next major version of Spark it should be fixed properly and
{{awaitTerminatio}} method should be declared to throw {{InterruptedException}}.
was (Author: vanekjar):
I think the suggested workaround is fine for the current 1.x version of Spark.
So updating the documentation would be proper solution to prevent other
developers from unexpected problems.
But in the next major version of Spark it should be fixed properly and
`awaitTermination` method should be declared to throw `InterruptedException`.
JavaStreamingContext#awaitTermination() throws non-declared
InterruptedException
Key: SPARK-8393
URL: https://issues.apache.org/jira/browse/SPARK-8393
Project: Spark
Issue Type: Bug
Components: Streaming
Affects Versions: 1.3.1
Reporter: Jaromir Vanek
Priority: Trivial
Call to {{JavaStreamingContext#awaitTermination()}} can throw
{{InterruptedException}} which cannot be caught easily in Java because it's
not declared in {{@throws(classOf[InterruptedException])}} annotation.
This {{InterruptedException}} comes originally from {{ContextWaiter}} where
Java {{ReentrantLock}} is used.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Comment Edited] (SPARK-8393) JavaStreamingContext#awaitTermination() throws non-declared InterruptedException
[
https://issues.apache.org/jira/browse/SPARK-8393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14599185#comment-14599185
]
Jaromir Vanek edited comment on SPARK-8393 at 6/24/15 9:49 AM:
---
I think the suggested workaround is fine for the current 1.x version of Spark.
So updating the documentation would be proper solution to prevent other
developers from unexpected problems.
But in the next major version of Spark it should be fixed properly and
{{awaitTermination}} method should be declared to throw
{{InterruptedException}}.
was (Author: vanekjar):
I think the suggested workaround is fine for the current 1.x version of Spark.
So updating the documentation would be proper solution to prevent other
developers from unexpected problems.
But in the next major version of Spark it should be fixed properly and
{{awaitTerminatio}} method should be declared to throw {{InterruptedException}}.
JavaStreamingContext#awaitTermination() throws non-declared
InterruptedException
Key: SPARK-8393
URL: https://issues.apache.org/jira/browse/SPARK-8393
Project: Spark
Issue Type: Bug
Components: Streaming
Affects Versions: 1.3.1
Reporter: Jaromir Vanek
Priority: Trivial
Call to {{JavaStreamingContext#awaitTermination()}} can throw
{{InterruptedException}} which cannot be caught easily in Java because it's
not declared in {{@throws(classOf[InterruptedException])}} annotation.
This {{InterruptedException}} comes originally from {{ContextWaiter}} where
Java {{ReentrantLock}} is used.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-8393) JavaStreamingContext#awaitTermination() throws non-declared InterruptedException
[
https://issues.apache.org/jira/browse/SPARK-8393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14599185#comment-14599185
]
Jaromir Vanek commented on SPARK-8393:
--
I think the suggested workaround is fine for the current 1.x version of Spark.
So updating the documentation would be proper solution to prevent other
developers from unexpected problems.
But in the next major version of Spark it should be fixed properly and
`awaitTermination` method should be declared to throw `InterruptedException`.
JavaStreamingContext#awaitTermination() throws non-declared
InterruptedException
Key: SPARK-8393
URL: https://issues.apache.org/jira/browse/SPARK-8393
Project: Spark
Issue Type: Bug
Components: Streaming
Affects Versions: 1.3.1
Reporter: Jaromir Vanek
Priority: Trivial
Call to {{JavaStreamingContext#awaitTermination()}} can throw
{{InterruptedException}} which cannot be caught easily in Java because it's
not declared in {{@throws(classOf[InterruptedException])}} annotation.
This {{InterruptedException}} comes originally from {{ContextWaiter}} where
Java {{ReentrantLock}} is used.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-8393) JavaStreamingContext#awaitTermination() throws non-declared InterruptedException
[
https://issues.apache.org/jira/browse/SPARK-8393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14589483#comment-14589483
]
Jaromir Vanek commented on SPARK-8393:
--
{{awaitTerminationOrTimeout}} also throws {{InterruptedException}} in case the
thread is interrupted, the only difference is the timeout feature.
JavaStreamingContext#awaitTermination() throws non-declared
InterruptedException
Key: SPARK-8393
URL: https://issues.apache.org/jira/browse/SPARK-8393
Project: Spark
Issue Type: Bug
Components: Streaming
Affects Versions: 1.3.1
Reporter: Jaromir Vanek
Priority: Trivial
Call to {{JavaStreamingContext#awaitTermination()}} can throw
{{InterruptedException}} which cannot be caught easily in Java because it's
not declared in {{@throws(classOf[InterruptedException])}} annotation.
This {{InterruptedException}} comes originally from {{ContextWaiter}} where
Java {{ReentrantLock}} is used.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-8393) JavaStreamingContext#awaitTermination() throws non-declared InterruptedException
[
https://issues.apache.org/jira/browse/SPARK-8393?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jaromir Vanek updated SPARK-8393:
-
Description:
Call to {{JavaStreamingContext#awaitTermination()}} can throw
InterruptedException which cannot be caught easily in Java because it's not
declared in {{@throws(classOf[InterruptedException])}} annotation.
This InterruptedException comes originally from ContextWaiter where Java
ReentrantLock is used.
was:
Call to JavaStreamingContext#awaitTermination() can throw InterruptedException
which cannot be caught easily in Java because it's not declared in
@throws(classOf[InterruptedException]) annotation.
This InterruptedException comes originally from ContextWaiter where Java
ReentrantLock is used.
JavaStreamingContext#awaitTermination() throws non-declared
InterruptedException
Key: SPARK-8393
URL: https://issues.apache.org/jira/browse/SPARK-8393
Project: Spark
Issue Type: Bug
Components: Streaming
Affects Versions: 1.3.1
Reporter: Jaromir Vanek
Priority: Trivial
Call to {{JavaStreamingContext#awaitTermination()}} can throw
InterruptedException which cannot be caught easily in Java because it's not
declared in {{@throws(classOf[InterruptedException])}} annotation.
This InterruptedException comes originally from ContextWaiter where Java
ReentrantLock is used.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-8393) JavaStreamingContext#awaitTermination() throws non-declared InterruptedException
[
https://issues.apache.org/jira/browse/SPARK-8393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14588013#comment-14588013
]
Jaromir Vanek commented on SPARK-8393:
--
It's not a big problem in Java. But I took me quite a bit of time to realize
where exactly this {{InterruptedException}} comes from.
In Java it can be caught as general {{Exception}}:
{code}
try {
streamingContext.awaitTermination();
} catch (Exception e) {
if (e instanceof InterruptedException) {
// handle exception
}
{code}
As far as I know {{awaitTerminationOrTimeout}} may throw the same exception as
well.
JavaStreamingContext#awaitTermination() throws non-declared
InterruptedException
Key: SPARK-8393
URL: https://issues.apache.org/jira/browse/SPARK-8393
Project: Spark
Issue Type: Bug
Components: Streaming
Affects Versions: 1.3.1
Reporter: Jaromir Vanek
Priority: Trivial
Call to JavaStreamingContext#awaitTermination() can throw
InterruptedException which cannot be caught easily in Java because it's not
declared in @throws(classOf[InterruptedException]) annotation.
This InterruptedException comes originally from ContextWaiter where Java
ReentrantLock is used.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-8393) JavaStreamingContext#awaitTermination() throws non-declared InterruptedException
[
https://issues.apache.org/jira/browse/SPARK-8393?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jaromir Vanek updated SPARK-8393:
-
Description:
Call to {{JavaStreamingContext#awaitTermination()}} can throw
{{InterruptedException}} which cannot be caught easily in Java because it's not
declared in {{@throws(classOf[InterruptedException])}} annotation.
This {{InterruptedException}} comes originally from {{ContextWaiter}} where
Java {{ReentrantLock}} is used.
was:
Call to {{JavaStreamingContext#awaitTermination()}} can throw
InterruptedException which cannot be caught easily in Java because it's not
declared in {{@throws(classOf[InterruptedException])}} annotation.
This {{InterruptedException}} comes originally from {{ContextWaiter}} where
Java {{ReentrantLock}} is used.
JavaStreamingContext#awaitTermination() throws non-declared
InterruptedException
Key: SPARK-8393
URL: https://issues.apache.org/jira/browse/SPARK-8393
Project: Spark
Issue Type: Bug
Components: Streaming
Affects Versions: 1.3.1
Reporter: Jaromir Vanek
Priority: Trivial
Call to {{JavaStreamingContext#awaitTermination()}} can throw
{{InterruptedException}} which cannot be caught easily in Java because it's
not declared in {{@throws(classOf[InterruptedException])}} annotation.
This {{InterruptedException}} comes originally from {{ContextWaiter}} where
Java {{ReentrantLock}} is used.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-8393) JavaStreamingContext#awaitTermination() throws non-declared InterruptedException
[
https://issues.apache.org/jira/browse/SPARK-8393?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jaromir Vanek updated SPARK-8393:
-
Description:
Call to {{JavaStreamingContext#awaitTermination()}} can throw
InterruptedException which cannot be caught easily in Java because it's not
declared in {{@throws(classOf[InterruptedException])}} annotation.
This {{InterruptedException}} comes originally from {{ContextWaiter}} where
Java {{ReentrantLock}} is used.
was:
Call to {{JavaStreamingContext#awaitTermination()}} can throw
InterruptedException which cannot be caught easily in Java because it's not
declared in {{@throws(classOf[InterruptedException])}} annotation.
This InterruptedException comes originally from ContextWaiter where Java
ReentrantLock is used.
JavaStreamingContext#awaitTermination() throws non-declared
InterruptedException
Key: SPARK-8393
URL: https://issues.apache.org/jira/browse/SPARK-8393
Project: Spark
Issue Type: Bug
Components: Streaming
Affects Versions: 1.3.1
Reporter: Jaromir Vanek
Priority: Trivial
Call to {{JavaStreamingContext#awaitTermination()}} can throw
InterruptedException which cannot be caught easily in Java because it's not
declared in {{@throws(classOf[InterruptedException])}} annotation.
This {{InterruptedException}} comes originally from {{ContextWaiter}} where
Java {{ReentrantLock}} is used.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-8393) JavaStreamingContext#awaitTermination() throws non-declared InterruptedException
[ https://issues.apache.org/jira/browse/SPARK-8393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14587822#comment-14587822 ] Jaromir Vanek commented on SPARK-8393: -- Call to JavaStreamingContext#awaitTermination() can throw InterruptedException which cannot be caught easily in Java because it's not declared in @throws(classOf[InterruptedException]) annotation. This InterruptedException comes originally from ContextWaiter where Java ReentrantLock is used. JavaStreamingContext#awaitTermination() throws non-declared InterruptedException Key: SPARK-8393 URL: https://issues.apache.org/jira/browse/SPARK-8393 Project: Spark Issue Type: Bug Components: Streaming Affects Versions: 1.3.1 Reporter: Jaromir Vanek Priority: Trivial -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-8393) JavaStreamingContext#awaitTermination() throws non-declared InterruptedException
[ https://issues.apache.org/jira/browse/SPARK-8393?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jaromir Vanek updated SPARK-8393: - Description: Call to JavaStreamingContext#awaitTermination() can throw InterruptedException which cannot be caught easily in Java because it's not declared in @throws(classOf[InterruptedException]) annotation. This InterruptedException comes originally from ContextWaiter where Java ReentrantLock is used. JavaStreamingContext#awaitTermination() throws non-declared InterruptedException Key: SPARK-8393 URL: https://issues.apache.org/jira/browse/SPARK-8393 Project: Spark Issue Type: Bug Components: Streaming Affects Versions: 1.3.1 Reporter: Jaromir Vanek Priority: Trivial Call to JavaStreamingContext#awaitTermination() can throw InterruptedException which cannot be caught easily in Java because it's not declared in @throws(classOf[InterruptedException]) annotation. This InterruptedException comes originally from ContextWaiter where Java ReentrantLock is used. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Issue Comment Deleted] (SPARK-8393) JavaStreamingContext#awaitTermination() throws non-declared InterruptedException
[ https://issues.apache.org/jira/browse/SPARK-8393?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jaromir Vanek updated SPARK-8393: - Comment: was deleted (was: Call to JavaStreamingContext#awaitTermination() can throw InterruptedException which cannot be caught easily in Java because it's not declared in @throws(classOf[InterruptedException]) annotation. This InterruptedException comes originally from ContextWaiter where Java ReentrantLock is used.) JavaStreamingContext#awaitTermination() throws non-declared InterruptedException Key: SPARK-8393 URL: https://issues.apache.org/jira/browse/SPARK-8393 Project: Spark Issue Type: Bug Components: Streaming Affects Versions: 1.3.1 Reporter: Jaromir Vanek Priority: Trivial Call to JavaStreamingContext#awaitTermination() can throw InterruptedException which cannot be caught easily in Java because it's not declared in @throws(classOf[InterruptedException]) annotation. This InterruptedException comes originally from ContextWaiter where Java ReentrantLock is used. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-8393) JavaStreamingContext#awaitTermination() throws non-declared InterruptedException
Jaromir Vanek created SPARK-8393: Summary: JavaStreamingContext#awaitTermination() throws non-declared InterruptedException Key: SPARK-8393 URL: https://issues.apache.org/jira/browse/SPARK-8393 Project: Spark Issue Type: Bug Components: Streaming Affects Versions: 1.3.1 Reporter: Jaromir Vanek Priority: Trivial -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
