[jira] [Commented] (SPARK-10568) Error thrown in stopping one component in SparkContext.stop() doesn't allow other components to be stopped
[
https://issues.apache.org/jira/browse/SPARK-10568?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14901382#comment-14901382
]
Matt Cheah commented on SPARK-10568:
Actually I think this was handled by
https://issues.apache.org/jira/browse/SPARK-9446 and its associated PR
https://github.com/apache/spark/pull/7756 and I didn't catch it because when I
filed this I was looking at the 1.4.1 release code.
Sorry for the inconvenient duplicate. I suppose that SparkEnv.stop() isn't
doing the same calls to Utils.tryLogNonFatalError for each of its components
though. Maybe we can make this JIRA about cleaning up what's remaining?
> Error thrown in stopping one component in SparkContext.stop() doesn't allow
> other components to be stopped
> --
>
> Key: SPARK-10568
> URL: https://issues.apache.org/jira/browse/SPARK-10568
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
>Affects Versions: 1.4.1
>Reporter: Matt Cheah
>Priority: Minor
>
> When I shut down a Java process that is running a SparkContext, it invokes a
> shutdown hook that eventually calls SparkContext.stop(), and inside
> SparkContext.stop() each individual component (DiskBlockManager, Scheduler
> Backend) is stopped. If an exception is thrown in stopping one of these
> components, none of the other components will be stopped cleanly either. This
> caused problems when I stopped a Java process running a Spark context in
> yarn-client mode, because not properly stopping YarnSchedulerBackend leads to
> problems.
> The steps I ran are as follows:
> 1. Create one job which fills the cluster
> 2. Kick off another job which creates a Spark Context
> 3. Kill the Java process with the Spark Context in #2
> 4. The job remains in the YARN UI as ACCEPTED
> Looking in the logs we see the following:
> {code}
> 2015-09-07 10:32:43,446 ERROR [Thread-3] o.a.s.u.Utils - Uncaught exception
> in thread Thread-3
> java.lang.NullPointerException: null
> at
> org.apache.spark.storage.DiskBlockManager.org$apache$spark$storage$DiskBlockManager$$doStop(DiskBlockManager.scala:162)
> ~[spark-core_2.10-1.4.1.jar:1.4.1]
> at
> org.apache.spark.storage.DiskBlockManager$$anonfun$addShutdownHook$1.apply$mcV$sp(DiskBlockManager.scala:144)
> ~[spark-core_2.10-1.4.1.jar:1.4.1]
> at org.apache.spark.util.SparkShutdownHook.run(Utils.scala:2308)
> ~[spark-core_2.10-1.4.1.jar:1.4.1]
> {code}
> I think what's going on is that when we kill the application in the queued
> state, it tries to run the SparkContext.stop() method on the driver and stop
> each component. It dies trying to stop the DiskBlockManager because it hasn't
> been initialized yet - the application is still waiting to be scheduled by
> the Yarn RM - but YarnClient.stop() is not invoked as a result, leaving the
> application sticking around in the accepted state.
> Because of what appears to be bugs in the YARN scheduler, entering this state
> makes it so that the YARN scheduler is unable to schedule any more jobs
> unless we manually remove this application via the YARN CLI. We can tackle
> the YARN stuck state separately, but ensuring that all components get at
> least some chance to stop when a SparkContext stops seems like a good idea.
> Of course we can still throw some exception and/or log exceptions for
> everything that goes wrong at the end of stopping the context.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-10568) Error thrown in stopping one component in SparkContext.stop() doesn't allow other components to be stopped
[
https://issues.apache.org/jira/browse/SPARK-10568?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14901164#comment-14901164
]
Matt Cheah commented on SPARK-10568:
Yup!
> Error thrown in stopping one component in SparkContext.stop() doesn't allow
> other components to be stopped
> --
>
> Key: SPARK-10568
> URL: https://issues.apache.org/jira/browse/SPARK-10568
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
>Affects Versions: 1.4.1
>Reporter: Matt Cheah
>Priority: Minor
>
> When I shut down a Java process that is running a SparkContext, it invokes a
> shutdown hook that eventually calls SparkContext.stop(), and inside
> SparkContext.stop() each individual component (DiskBlockManager, Scheduler
> Backend) is stopped. If an exception is thrown in stopping one of these
> components, none of the other components will be stopped cleanly either. This
> caused problems when I stopped a Java process running a Spark context in
> yarn-client mode, because not properly stopping YarnSchedulerBackend leads to
> problems.
> The steps I ran are as follows:
> 1. Create one job which fills the cluster
> 2. Kick off another job which creates a Spark Context
> 3. Kill the Java process with the Spark Context in #2
> 4. The job remains in the YARN UI as ACCEPTED
> Looking in the logs we see the following:
> {code}
> 2015-09-07 10:32:43,446 ERROR [Thread-3] o.a.s.u.Utils - Uncaught exception
> in thread Thread-3
> java.lang.NullPointerException: null
> at
> org.apache.spark.storage.DiskBlockManager.org$apache$spark$storage$DiskBlockManager$$doStop(DiskBlockManager.scala:162)
> ~[spark-core_2.10-1.4.1.jar:1.4.1]
> at
> org.apache.spark.storage.DiskBlockManager$$anonfun$addShutdownHook$1.apply$mcV$sp(DiskBlockManager.scala:144)
> ~[spark-core_2.10-1.4.1.jar:1.4.1]
> at org.apache.spark.util.SparkShutdownHook.run(Utils.scala:2308)
> ~[spark-core_2.10-1.4.1.jar:1.4.1]
> {code}
> I think what's going on is that when we kill the application in the queued
> state, it tries to run the SparkContext.stop() method on the driver and stop
> each component. It dies trying to stop the DiskBlockManager because it hasn't
> been initialized yet - the application is still waiting to be scheduled by
> the Yarn RM - but YarnClient.stop() is not invoked as a result, leaving the
> application sticking around in the accepted state.
> Because of what appears to be bugs in the YARN scheduler, entering this state
> makes it so that the YARN scheduler is unable to schedule any more jobs
> unless we manually remove this application via the YARN CLI. We can tackle
> the YARN stuck state separately, but ensuring that all components get at
> least some chance to stop when a SparkContext stops seems like a good idea.
> Of course we can still throw some exception and/or log exceptions for
> everything that goes wrong at the end of stopping the context.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-10568) Error thrown in stopping one component in SparkContext.stop() doesn't allow other components to be stopped
[
https://issues.apache.org/jira/browse/SPARK-10568?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14901396#comment-14901396
]
Sean Owen commented on SPARK-10568:
---
Sure, if you have suggestions along those lines. I had imagined it was about
calling some code to stop each component in a way that exceptions from one
would not prevent the rest from executing.
> Error thrown in stopping one component in SparkContext.stop() doesn't allow
> other components to be stopped
> --
>
> Key: SPARK-10568
> URL: https://issues.apache.org/jira/browse/SPARK-10568
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
>Affects Versions: 1.4.1
>Reporter: Matt Cheah
>Priority: Minor
>
> When I shut down a Java process that is running a SparkContext, it invokes a
> shutdown hook that eventually calls SparkContext.stop(), and inside
> SparkContext.stop() each individual component (DiskBlockManager, Scheduler
> Backend) is stopped. If an exception is thrown in stopping one of these
> components, none of the other components will be stopped cleanly either. This
> caused problems when I stopped a Java process running a Spark context in
> yarn-client mode, because not properly stopping YarnSchedulerBackend leads to
> problems.
> The steps I ran are as follows:
> 1. Create one job which fills the cluster
> 2. Kick off another job which creates a Spark Context
> 3. Kill the Java process with the Spark Context in #2
> 4. The job remains in the YARN UI as ACCEPTED
> Looking in the logs we see the following:
> {code}
> 2015-09-07 10:32:43,446 ERROR [Thread-3] o.a.s.u.Utils - Uncaught exception
> in thread Thread-3
> java.lang.NullPointerException: null
> at
> org.apache.spark.storage.DiskBlockManager.org$apache$spark$storage$DiskBlockManager$$doStop(DiskBlockManager.scala:162)
> ~[spark-core_2.10-1.4.1.jar:1.4.1]
> at
> org.apache.spark.storage.DiskBlockManager$$anonfun$addShutdownHook$1.apply$mcV$sp(DiskBlockManager.scala:144)
> ~[spark-core_2.10-1.4.1.jar:1.4.1]
> at org.apache.spark.util.SparkShutdownHook.run(Utils.scala:2308)
> ~[spark-core_2.10-1.4.1.jar:1.4.1]
> {code}
> I think what's going on is that when we kill the application in the queued
> state, it tries to run the SparkContext.stop() method on the driver and stop
> each component. It dies trying to stop the DiskBlockManager because it hasn't
> been initialized yet - the application is still waiting to be scheduled by
> the Yarn RM - but YarnClient.stop() is not invoked as a result, leaving the
> application sticking around in the accepted state.
> Because of what appears to be bugs in the YARN scheduler, entering this state
> makes it so that the YARN scheduler is unable to schedule any more jobs
> unless we manually remove this application via the YARN CLI. We can tackle
> the YARN stuck state separately, but ensuring that all components get at
> least some chance to stop when a SparkContext stops seems like a good idea.
> Of course we can still throw some exception and/or log exceptions for
> everything that goes wrong at the end of stopping the context.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-10568) Error thrown in stopping one component in SparkContext.stop() doesn't allow other components to be stopped
[
https://issues.apache.org/jira/browse/SPARK-10568?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14877037#comment-14877037
]
Sean Owen commented on SPARK-10568:
---
[~mcheah] are you working on a PR for this?
> Error thrown in stopping one component in SparkContext.stop() doesn't allow
> other components to be stopped
> --
>
> Key: SPARK-10568
> URL: https://issues.apache.org/jira/browse/SPARK-10568
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
>Affects Versions: 1.4.1
>Reporter: Matt Cheah
>Priority: Minor
>
> When I shut down a Java process that is running a SparkContext, it invokes a
> shutdown hook that eventually calls SparkContext.stop(), and inside
> SparkContext.stop() each individual component (DiskBlockManager, Scheduler
> Backend) is stopped. If an exception is thrown in stopping one of these
> components, none of the other components will be stopped cleanly either. This
> caused problems when I stopped a Java process running a Spark context in
> yarn-client mode, because not properly stopping YarnSchedulerBackend leads to
> problems.
> The steps I ran are as follows:
> 1. Create one job which fills the cluster
> 2. Kick off another job which creates a Spark Context
> 3. Kill the Java process with the Spark Context in #2
> 4. The job remains in the YARN UI as ACCEPTED
> Looking in the logs we see the following:
> {code}
> 2015-09-07 10:32:43,446 ERROR [Thread-3] o.a.s.u.Utils - Uncaught exception
> in thread Thread-3
> java.lang.NullPointerException: null
> at
> org.apache.spark.storage.DiskBlockManager.org$apache$spark$storage$DiskBlockManager$$doStop(DiskBlockManager.scala:162)
> ~[spark-core_2.10-1.4.1.jar:1.4.1]
> at
> org.apache.spark.storage.DiskBlockManager$$anonfun$addShutdownHook$1.apply$mcV$sp(DiskBlockManager.scala:144)
> ~[spark-core_2.10-1.4.1.jar:1.4.1]
> at org.apache.spark.util.SparkShutdownHook.run(Utils.scala:2308)
> ~[spark-core_2.10-1.4.1.jar:1.4.1]
> {code}
> I think what's going on is that when we kill the application in the queued
> state, it tries to run the SparkContext.stop() method on the driver and stop
> each component. It dies trying to stop the DiskBlockManager because it hasn't
> been initialized yet - the application is still waiting to be scheduled by
> the Yarn RM - but YarnClient.stop() is not invoked as a result, leaving the
> application sticking around in the accepted state.
> Because of what appears to be bugs in the YARN scheduler, entering this state
> makes it so that the YARN scheduler is unable to schedule any more jobs
> unless we manually remove this application via the YARN CLI. We can tackle
> the YARN stuck state separately, but ensuring that all components get at
> least some chance to stop when a SparkContext stops seems like a good idea.
> Of course we can still throw some exception and/or log exceptions for
> everything that goes wrong at the end of stopping the context.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-10568) Error thrown in stopping one component in SparkContext.stop() doesn't allow other components to be stopped
[
https://issues.apache.org/jira/browse/SPARK-10568?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14741978#comment-14741978
]
Sean Owen commented on SPARK-10568:
---
Yeah I can imagine some relatively painless Scala code that iterates over a
bunch of closures that shutdown things and log exceptions and continue in the
face of failures. Ideally we check if shutdown happens in the reverse order of
initialization too.
OK if the YARN issue is really outside Spark yes that's out of scope here.
> Error thrown in stopping one component in SparkContext.stop() doesn't allow
> other components to be stopped
> --
>
> Key: SPARK-10568
> URL: https://issues.apache.org/jira/browse/SPARK-10568
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
>Affects Versions: 1.4.1
>Reporter: Matt Cheah
>
> When I shut down a Java process that is running a SparkContext, it invokes a
> shutdown hook that eventually calls SparkContext.stop(), and inside
> SparkContext.stop() each individual component (DiskBlockManager, Scheduler
> Backend) is stopped. If an exception is thrown in stopping one of these
> components, none of the other components will be stopped cleanly either. This
> caused problems when I stopped a Java process running a Spark context in
> yarn-client mode, because not properly stopping YarnSchedulerBackend leads to
> problems.
> The steps I ran are as follows:
> 1. Create one job which fills the cluster
> 2. Kick off another job which creates a Spark Context
> 3. Kill the Java process with the Spark Context in #2
> 4. The job remains in the YARN UI as ACCEPTED
> Looking in the logs we see the following:
> {code}
> 2015-09-07 10:32:43,446 ERROR [Thread-3] o.a.s.u.Utils - Uncaught exception
> in thread Thread-3
> java.lang.NullPointerException: null
> at
> org.apache.spark.storage.DiskBlockManager.org$apache$spark$storage$DiskBlockManager$$doStop(DiskBlockManager.scala:162)
> ~[spark-core_2.10-1.4.1.jar:1.4.1]
> at
> org.apache.spark.storage.DiskBlockManager$$anonfun$addShutdownHook$1.apply$mcV$sp(DiskBlockManager.scala:144)
> ~[spark-core_2.10-1.4.1.jar:1.4.1]
> at org.apache.spark.util.SparkShutdownHook.run(Utils.scala:2308)
> ~[spark-core_2.10-1.4.1.jar:1.4.1]
> {code}
> I think what's going on is that when we kill the application in the queued
> state, it tries to run the SparkContext.stop() method on the driver and stop
> each component. It dies trying to stop the DiskBlockManager because it hasn't
> been initialized yet - the application is still waiting to be scheduled by
> the Yarn RM - but YarnClient.stop() is not invoked as a result, leaving the
> application sticking around in the accepted state.
> Because of what appears to be bugs in the YARN scheduler, entering this state
> makes it so that the YARN scheduler is unable to schedule any more jobs
> unless we manually remove this application via the YARN CLI. We can tackle
> the YARN stuck state separately, but ensuring that all components get at
> least some chance to stop when a SparkContext stops seems like a good idea.
> Of course we can still throw some exception and/or log exceptions for
> everything that goes wrong at the end of stopping the context.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-10568) Error thrown in stopping one component in SparkContext.stop() doesn't allow other components to be stopped
[
https://issues.apache.org/jira/browse/SPARK-10568?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14741526#comment-14741526
]
Matt Cheah commented on SPARK-10568:
Also I think the YARN issue is something more suited for discussing in the
Apache YARN project.
> Error thrown in stopping one component in SparkContext.stop() doesn't allow
> other components to be stopped
> --
>
> Key: SPARK-10568
> URL: https://issues.apache.org/jira/browse/SPARK-10568
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
>Affects Versions: 1.4.1
>Reporter: Matt Cheah
>
> When I shut down a Java process that is running a SparkContext, it invokes a
> shutdown hook that eventually calls SparkContext.stop(), and inside
> SparkContext.stop() each individual component (DiskBlockManager, Scheduler
> Backend) is stopped. If an exception is thrown in stopping one of these
> components, none of the other components will be stopped cleanly either. This
> caused problems when I stopped a Java process running a Spark context in
> yarn-client mode, because not properly stopping YarnSchedulerBackend leads to
> problems.
> The steps I ran are as follows:
> 1. Create one job which fills the cluster
> 2. Kick off another job which creates a Spark Context
> 3. Kill the Java process with the Spark Context in #2
> 4. The job remains in the YARN UI as ACCEPTED
> Looking in the logs we see the following:
> {code}
> 2015-09-07 10:32:43,446 ERROR [Thread-3] o.a.s.u.Utils - Uncaught exception
> in thread Thread-3
> java.lang.NullPointerException: null
> at
> org.apache.spark.storage.DiskBlockManager.org$apache$spark$storage$DiskBlockManager$$doStop(DiskBlockManager.scala:162)
> ~[spark-core_2.10-1.4.1.jar:1.4.1]
> at
> org.apache.spark.storage.DiskBlockManager$$anonfun$addShutdownHook$1.apply$mcV$sp(DiskBlockManager.scala:144)
> ~[spark-core_2.10-1.4.1.jar:1.4.1]
> at org.apache.spark.util.SparkShutdownHook.run(Utils.scala:2308)
> ~[spark-core_2.10-1.4.1.jar:1.4.1]
> {code}
> I think what's going on is that when we kill the application in the queued
> state, it tries to run the SparkContext.stop() method on the driver and stop
> each component. It dies trying to stop the DiskBlockManager because it hasn't
> been initialized yet - the application is still waiting to be scheduled by
> the Yarn RM - but YarnClient.stop() is not invoked as a result, leaving the
> application sticking around in the accepted state.
> Because of what appears to be bugs in the YARN scheduler, entering this state
> makes it so that the YARN scheduler is unable to schedule any more jobs
> unless we manually remove this application via the YARN CLI. We can tackle
> the YARN stuck state separately, but ensuring that all components get at
> least some chance to stop when a SparkContext stops seems like a good idea.
> Of course we can still throw some exception and/or log exceptions for
> everything that goes wrong at the end of stopping the context.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-10568) Error thrown in stopping one component in SparkContext.stop() doesn't allow other components to be stopped
[
https://issues.apache.org/jira/browse/SPARK-10568?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14741488#comment-14741488
]
Sean Owen commented on SPARK-10568:
---
I think this is a duplicate of SPARK-10554? at least, the immediate issue is.
It's tough to know whether to fail fast or keep trying operations of other
things in a failed state, which might just create more errors.
It might be most effective to identify common, impactful cases where shutdown
needs to more reliably clean up. SPARK-10554 is one example and sounds like you
have another YARN example. Maybe this should focus on the YARN issue?
> Error thrown in stopping one component in SparkContext.stop() doesn't allow
> other components to be stopped
> --
>
> Key: SPARK-10568
> URL: https://issues.apache.org/jira/browse/SPARK-10568
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
>Affects Versions: 1.4.1
>Reporter: Matt Cheah
>
> When I shut down a Java process that is running a SparkContext, it invokes a
> shutdown hook that eventually calls SparkContext.stop(), and inside
> SparkContext.stop() each individual component (DiskBlockManager, Scheduler
> Backend) is stopped. If an exception is thrown in stopping one of these
> components, none of the other components will be stopped cleanly either. This
> caused problems when I stopped a Java process running a Spark context in
> yarn-client mode, because not properly stopping YarnSchedulerBackend leads to
> problems.
> The steps I ran are as follows:
> 1. Create one job which fills the cluster
> 2. Kick off another job which creates a Spark Context
> 3. Kill the Java process with the Spark Context in #2
> 4. The job remains in the YARN UI as ACCEPTED
> Looking in the logs we see the following:
> {code}
> 2015-09-07 10:32:43,446 ERROR [Thread-3] o.a.s.u.Utils - Uncaught exception
> in thread Thread-3
> java.lang.NullPointerException: null
> at
> org.apache.spark.storage.DiskBlockManager.org$apache$spark$storage$DiskBlockManager$$doStop(DiskBlockManager.scala:162)
> ~[spark-core_2.10-1.4.1.jar:1.4.1]
> at
> org.apache.spark.storage.DiskBlockManager$$anonfun$addShutdownHook$1.apply$mcV$sp(DiskBlockManager.scala:144)
> ~[spark-core_2.10-1.4.1.jar:1.4.1]
> at org.apache.spark.util.SparkShutdownHook.run(Utils.scala:2308)
> ~[spark-core_2.10-1.4.1.jar:1.4.1]
> {code}
> I think what's going on is that when we kill the application in the queued
> state, it tries to run the SparkContext.stop() method on the driver and stop
> each component. It dies trying to stop the DiskBlockManager because it hasn't
> been initialized yet - the application is still waiting to be scheduled by
> the Yarn RM - but YarnClient.stop() is not invoked as a result, leaving the
> application sticking around in the accepted state.
> Because of what appears to be bugs in the YARN scheduler, entering this state
> makes it so that the YARN scheduler is unable to schedule any more jobs
> unless we manually remove this application via the YARN CLI. We can tackle
> the YARN stuck state separately, but ensuring that all components get at
> least some chance to stop when a SparkContext stops seems like a good idea.
> Of course we can still throw some exception and/or log exceptions for
> everything that goes wrong at the end of stopping the context.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-10568) Error thrown in stopping one component in SparkContext.stop() doesn't allow other components to be stopped
[
https://issues.apache.org/jira/browse/SPARK-10568?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14741525#comment-14741525
]
Matt Cheah commented on SPARK-10568:
I'm more concerned about the general case which is that failing in one part of
stop() doesn't allow for stopping anything else. We should fix the NPE in
DiskBlockManager, yes, but we should probably make stopping more robust in the
sense of at least trying to stop each component.
So we need to fix DiskBlockManager, yes, but we also should look at handling
uncaught exceptions from stopping each component more carefully.
> Error thrown in stopping one component in SparkContext.stop() doesn't allow
> other components to be stopped
> --
>
> Key: SPARK-10568
> URL: https://issues.apache.org/jira/browse/SPARK-10568
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
>Affects Versions: 1.4.1
>Reporter: Matt Cheah
>
> When I shut down a Java process that is running a SparkContext, it invokes a
> shutdown hook that eventually calls SparkContext.stop(), and inside
> SparkContext.stop() each individual component (DiskBlockManager, Scheduler
> Backend) is stopped. If an exception is thrown in stopping one of these
> components, none of the other components will be stopped cleanly either. This
> caused problems when I stopped a Java process running a Spark context in
> yarn-client mode, because not properly stopping YarnSchedulerBackend leads to
> problems.
> The steps I ran are as follows:
> 1. Create one job which fills the cluster
> 2. Kick off another job which creates a Spark Context
> 3. Kill the Java process with the Spark Context in #2
> 4. The job remains in the YARN UI as ACCEPTED
> Looking in the logs we see the following:
> {code}
> 2015-09-07 10:32:43,446 ERROR [Thread-3] o.a.s.u.Utils - Uncaught exception
> in thread Thread-3
> java.lang.NullPointerException: null
> at
> org.apache.spark.storage.DiskBlockManager.org$apache$spark$storage$DiskBlockManager$$doStop(DiskBlockManager.scala:162)
> ~[spark-core_2.10-1.4.1.jar:1.4.1]
> at
> org.apache.spark.storage.DiskBlockManager$$anonfun$addShutdownHook$1.apply$mcV$sp(DiskBlockManager.scala:144)
> ~[spark-core_2.10-1.4.1.jar:1.4.1]
> at org.apache.spark.util.SparkShutdownHook.run(Utils.scala:2308)
> ~[spark-core_2.10-1.4.1.jar:1.4.1]
> {code}
> I think what's going on is that when we kill the application in the queued
> state, it tries to run the SparkContext.stop() method on the driver and stop
> each component. It dies trying to stop the DiskBlockManager because it hasn't
> been initialized yet - the application is still waiting to be scheduled by
> the Yarn RM - but YarnClient.stop() is not invoked as a result, leaving the
> application sticking around in the accepted state.
> Because of what appears to be bugs in the YARN scheduler, entering this state
> makes it so that the YARN scheduler is unable to schedule any more jobs
> unless we manually remove this application via the YARN CLI. We can tackle
> the YARN stuck state separately, but ensuring that all components get at
> least some chance to stop when a SparkContext stops seems like a good idea.
> Of course we can still throw some exception and/or log exceptions for
> everything that goes wrong at the end of stopping the context.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-10568) Error thrown in stopping one component in SparkContext.stop() doesn't allow other components to be stopped
[
https://issues.apache.org/jira/browse/SPARK-10568?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14741621#comment-14741621
]
Jian Feng Zhang commented on SPARK-10568:
-
Make sense to ensure each individual component is stopped in
SparkContext.stop().
> Error thrown in stopping one component in SparkContext.stop() doesn't allow
> other components to be stopped
> --
>
> Key: SPARK-10568
> URL: https://issues.apache.org/jira/browse/SPARK-10568
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
>Affects Versions: 1.4.1
>Reporter: Matt Cheah
>
> When I shut down a Java process that is running a SparkContext, it invokes a
> shutdown hook that eventually calls SparkContext.stop(), and inside
> SparkContext.stop() each individual component (DiskBlockManager, Scheduler
> Backend) is stopped. If an exception is thrown in stopping one of these
> components, none of the other components will be stopped cleanly either. This
> caused problems when I stopped a Java process running a Spark context in
> yarn-client mode, because not properly stopping YarnSchedulerBackend leads to
> problems.
> The steps I ran are as follows:
> 1. Create one job which fills the cluster
> 2. Kick off another job which creates a Spark Context
> 3. Kill the Java process with the Spark Context in #2
> 4. The job remains in the YARN UI as ACCEPTED
> Looking in the logs we see the following:
> {code}
> 2015-09-07 10:32:43,446 ERROR [Thread-3] o.a.s.u.Utils - Uncaught exception
> in thread Thread-3
> java.lang.NullPointerException: null
> at
> org.apache.spark.storage.DiskBlockManager.org$apache$spark$storage$DiskBlockManager$$doStop(DiskBlockManager.scala:162)
> ~[spark-core_2.10-1.4.1.jar:1.4.1]
> at
> org.apache.spark.storage.DiskBlockManager$$anonfun$addShutdownHook$1.apply$mcV$sp(DiskBlockManager.scala:144)
> ~[spark-core_2.10-1.4.1.jar:1.4.1]
> at org.apache.spark.util.SparkShutdownHook.run(Utils.scala:2308)
> ~[spark-core_2.10-1.4.1.jar:1.4.1]
> {code}
> I think what's going on is that when we kill the application in the queued
> state, it tries to run the SparkContext.stop() method on the driver and stop
> each component. It dies trying to stop the DiskBlockManager because it hasn't
> been initialized yet - the application is still waiting to be scheduled by
> the Yarn RM - but YarnClient.stop() is not invoked as a result, leaving the
> application sticking around in the accepted state.
> Because of what appears to be bugs in the YARN scheduler, entering this state
> makes it so that the YARN scheduler is unable to schedule any more jobs
> unless we manually remove this application via the YARN CLI. We can tackle
> the YARN stuck state separately, but ensuring that all components get at
> least some chance to stop when a SparkContext stops seems like a good idea.
> Of course we can still throw some exception and/or log exceptions for
> everything that goes wrong at the end of stopping the context.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
