[jira] [Commented] (SPARK-3090) Avoid not stopping SparkContext with YARN Client mode
[ https://issues.apache.org/jira/browse/SPARK-3090?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14512048#comment-14512048 ] Apache Spark commented on SPARK-3090: - User 'vanzin' has created a pull request for this issue: https://github.com/apache/spark/pull/5696 Avoid not stopping SparkContext with YARN Client mode -- Key: SPARK-3090 URL: https://issues.apache.org/jira/browse/SPARK-3090 Project: Spark Issue Type: Bug Components: Spark Core, YARN Affects Versions: 1.1.0 Reporter: Kousuke Saruta When we use YARN Cluster mode, ApplicationMaser register a shutdown hook, stopping SparkContext. Thanks to this, SparkContext can stop even if Application forgets to stop SparkContext itself. But, unfortunately, YARN Client mode doesn't have such mechanism. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-3090) Avoid not stopping SparkContext with YARN Client mode
[ https://issues.apache.org/jira/browse/SPARK-3090?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14117036#comment-14117036 ] Kousuke Saruta commented on SPARK-3090: --- It sound good idea that SparkContext register shutdown-hook itself. Avoid not stopping SparkContext with YARN Client mode -- Key: SPARK-3090 URL: https://issues.apache.org/jira/browse/SPARK-3090 Project: Spark Issue Type: Bug Components: Spark Core, YARN Affects Versions: 1.1.0 Reporter: Kousuke Saruta When we use YARN Cluster mode, ApplicationMaser register a shutdown hook, stopping SparkContext. Thanks to this, SparkContext can stop even if Application forgets to stop SparkContext itself. But, unfortunately, YARN Client mode doesn't have such mechanism. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-3090) Avoid not stopping SparkContext with YARN Client mode
[ https://issues.apache.org/jira/browse/SPARK-3090?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14107488#comment-14107488 ] Marcelo Vanzin commented on SPARK-3090: --- I think that if we want to add this, it would be better to do so for all modes, not just yarn-client. Basically have SparkContext itself register a shutdown hook to shut itself down, and publish the priority of the hook so that apps / backends can register hooks that run before it (see Hadoop's ShutdownHookManager for the priority thing - http://goo.gl/BQ1bjk). That way the code in the yarn-cluster backend can be removed too. Avoid not stopping SparkContext with YARN Client mode -- Key: SPARK-3090 URL: https://issues.apache.org/jira/browse/SPARK-3090 Project: Spark Issue Type: Bug Components: Spark Core, YARN Affects Versions: 1.1.0 Reporter: Kousuke Saruta When we use YARN Cluster mode, ApplicationMaser register a shutdown hook, stopping SparkContext. Thanks to this, SparkContext can stop even if Application forgets to stop SparkContext itself. But, unfortunately, YARN Client mode doesn't have such mechanism. -- This message was sent by Atlassian JIRA (v6.2#6252) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
