[jira] [Commented] (SPARK-3732) Yarn Client: Add option to NOT System.exit() at end of main()
[ https://issues.apache.org/jira/browse/SPARK-3732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14152291#comment-14152291 ] Apache Spark commented on SPARK-3732: - User 'smatzana' has created a pull request for this issue: https://github.com/apache/spark/pull/2584 > Yarn Client: Add option to NOT System.exit() at end of main() > - > > Key: SPARK-3732 > URL: https://issues.apache.org/jira/browse/SPARK-3732 > Project: Spark > Issue Type: Improvement >Affects Versions: 1.1.0 >Reporter: Sotos Matzanas > Original Estimate: 1h > Remaining Estimate: 1h > > We would like to add the ability to create and submit Spark jobs > programmatically via Scala/Java. We have found a way to hack this and submit > jobs via Yarn, but since > org.apache.spark.deploy.yarn.Client.main() > exits with either 0 or 1 in the end, this will mean exit of our own program. > We would like to add an optional spark conf param to NOT exit at the end of > the main -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-3732) Yarn Client: Add option to NOT System.exit() at end of main()
[ https://issues.apache.org/jira/browse/SPARK-3732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14152294#comment-14152294 ] Marcelo Vanzin commented on SPARK-3732: --- I think that explicit System.exit() could just be removed. Exposing an option for this sounds like overkill. [~tgraves] had some comments about that call in the past, though. > Yarn Client: Add option to NOT System.exit() at end of main() > - > > Key: SPARK-3732 > URL: https://issues.apache.org/jira/browse/SPARK-3732 > Project: Spark > Issue Type: Improvement >Affects Versions: 1.1.0 >Reporter: Sotos Matzanas > Original Estimate: 1h > Remaining Estimate: 1h > > We would like to add the ability to create and submit Spark jobs > programmatically via Scala/Java. We have found a way to hack this and submit > jobs via Yarn, but since > org.apache.spark.deploy.yarn.Client.main() > exits with either 0 or 1 in the end, this will mean exit of our own program. > We would like to add an optional spark conf param to NOT exit at the end of > the main -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-3732) Yarn Client: Add option to NOT System.exit() at end of main()
[ https://issues.apache.org/jira/browse/SPARK-3732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14152302#comment-14152302 ] Sotos Matzanas commented on SPARK-3732: --- we added the option as insurance against backward compatibility, removing the System.exit() call will obviously work unless somebody is checking the exit code from spark-submit explicitely > Yarn Client: Add option to NOT System.exit() at end of main() > - > > Key: SPARK-3732 > URL: https://issues.apache.org/jira/browse/SPARK-3732 > Project: Spark > Issue Type: Improvement >Affects Versions: 1.1.0 >Reporter: Sotos Matzanas > Original Estimate: 1h > Remaining Estimate: 1h > > We would like to add the ability to create and submit Spark jobs > programmatically via Scala/Java. We have found a way to hack this and submit > jobs via Yarn, but since > org.apache.spark.deploy.yarn.Client.main() > exits with either 0 or 1 in the end, this will mean exit of our own program. > We would like to add an optional spark conf param to NOT exit at the end of > the main -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-3732) Yarn Client: Add option to NOT System.exit() at end of main()
[ https://issues.apache.org/jira/browse/SPARK-3732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14152308#comment-14152308 ] Thomas Graves commented on SPARK-3732: -- I think you should just change the name of this jira to add support for programmatically calling the spark yarn Client. As you have found this isn't currently supported and I wouldn't want to just remove the exit and say its supported without thinking about other implications. > Yarn Client: Add option to NOT System.exit() at end of main() > - > > Key: SPARK-3732 > URL: https://issues.apache.org/jira/browse/SPARK-3732 > Project: Spark > Issue Type: Improvement >Affects Versions: 1.1.0 >Reporter: Sotos Matzanas > Original Estimate: 1h > Remaining Estimate: 1h > > We would like to add the ability to create and submit Spark jobs > programmatically via Scala/Java. We have found a way to hack this and submit > jobs via Yarn, but since > org.apache.spark.deploy.yarn.Client.main() > exits with either 0 or 1 in the end, this will mean exit of our own program. > We would like to add an optional spark conf param to NOT exit at the end of > the main -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-3732) Yarn Client: Add option to NOT System.exit() at end of main()
[ https://issues.apache.org/jira/browse/SPARK-3732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14152309#comment-14152309 ] Marcelo Vanzin commented on SPARK-3732: --- Removing the call should work regardless; it's redundant, since the code will just exit normally anyway after that. > Yarn Client: Add option to NOT System.exit() at end of main() > - > > Key: SPARK-3732 > URL: https://issues.apache.org/jira/browse/SPARK-3732 > Project: Spark > Issue Type: Improvement >Affects Versions: 1.1.0 >Reporter: Sotos Matzanas > Original Estimate: 1h > Remaining Estimate: 1h > > We would like to add the ability to create and submit Spark jobs > programmatically via Scala/Java. We have found a way to hack this and submit > jobs via Yarn, but since > org.apache.spark.deploy.yarn.Client.main() > exits with either 0 or 1 in the end, this will mean exit of our own program. > We would like to add an optional spark conf param to NOT exit at the end of > the main -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-3732) Yarn Client: Add option to NOT System.exit() at end of main()
[
https://issues.apache.org/jira/browse/SPARK-3732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14152318#comment-14152318
]
Marcelo Vanzin commented on SPARK-3732:
---
BTW, if the call is removed, it should be possible to do what you want more
generically by calling {{SparkSubmit.main}} directly. That's still a little
fishy, but it's a lot less fishy than calling Yarn-specific code directly.
> Yarn Client: Add option to NOT System.exit() at end of main()
> -
>
> Key: SPARK-3732
> URL: https://issues.apache.org/jira/browse/SPARK-3732
> Project: Spark
> Issue Type: Improvement
>Affects Versions: 1.1.0
>Reporter: Sotos Matzanas
> Original Estimate: 1h
> Remaining Estimate: 1h
>
> We would like to add the ability to create and submit Spark jobs
> programmatically via Scala/Java. We have found a way to hack this and submit
> jobs via Yarn, but since
> org.apache.spark.deploy.yarn.Client.main()
> exits with either 0 or 1 in the end, this will mean exit of our own program.
> We would like to add an optional spark conf param to NOT exit at the end of
> the main
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-3732) Yarn Client: Add option to NOT System.exit() at end of main()
[ https://issues.apache.org/jira/browse/SPARK-3732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14152323#comment-14152323 ] Sotos Matzanas commented on SPARK-3732: --- [~tgraves] this jira is the first step for us to move forward with our own (very limited for now) version of programmatic job submits. I can add another jira to address the big issue, but we would like to see this one resolved first. Once we are confident with our solution we can contribute to the second jira. Let us know what you think > Yarn Client: Add option to NOT System.exit() at end of main() > - > > Key: SPARK-3732 > URL: https://issues.apache.org/jira/browse/SPARK-3732 > Project: Spark > Issue Type: Improvement >Affects Versions: 1.1.0 >Reporter: Sotos Matzanas > Original Estimate: 1h > Remaining Estimate: 1h > > We would like to add the ability to create and submit Spark jobs > programmatically via Scala/Java. We have found a way to hack this and submit > jobs via Yarn, but since > org.apache.spark.deploy.yarn.Client.main() > exits with either 0 or 1 in the end, this will mean exit of our own program. > We would like to add an optional spark conf param to NOT exit at the end of > the main -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-3732) Yarn Client: Add option to NOT System.exit() at end of main()
[ https://issues.apache.org/jira/browse/SPARK-3732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14152409#comment-14152409 ] Thomas Graves commented on SPARK-3732: -- I understand your usecase and need for it, but I think at this point I don't think we want to say we support it without properly addressing the bigger picture. The only public supported non-deprecated api is via spark-submit script. This means that we won't guarantee backwards compatibility on it. Note that we specifically discussed the Client beyond public on another PR and it was decided that it isn't officially supported. The only reason the object was left public was for backwards compatibility with how you used to start spark on yarn with the spark-class script. > Yarn Client: Add option to NOT System.exit() at end of main() > - > > Key: SPARK-3732 > URL: https://issues.apache.org/jira/browse/SPARK-3732 > Project: Spark > Issue Type: Improvement >Affects Versions: 1.1.0 >Reporter: Sotos Matzanas > Original Estimate: 1h > Remaining Estimate: 1h > > We would like to add the ability to create and submit Spark jobs > programmatically via Scala/Java. We have found a way to hack this and submit > jobs via Yarn, but since > org.apache.spark.deploy.yarn.Client.main() > exits with either 0 or 1 in the end, this will mean exit of our own program. > We would like to add an optional spark conf param to NOT exit at the end of > the main -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-3732) Yarn Client: Add option to NOT System.exit() at end of main()
[ https://issues.apache.org/jira/browse/SPARK-3732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14152912#comment-14152912 ] Sean Owen commented on SPARK-3732: -- FWIW, I was also surprised in the past that there is no way to submit a job programmatically. That would be great for embedding Spark. An option seems like big overkill here. System.exit() is not a great idea in general and I agree that removing it is better. I can confirm that the JVM exit status is 0 on success and 1 on exception anyway, so this doesn't even change semantics. That is, you still get exit status 1 if an exception is thrown. The stack trace is also printed, so the printing the exception is also redundant and the try block can go. > Yarn Client: Add option to NOT System.exit() at end of main() > - > > Key: SPARK-3732 > URL: https://issues.apache.org/jira/browse/SPARK-3732 > Project: Spark > Issue Type: Improvement >Affects Versions: 1.1.0 >Reporter: Sotos Matzanas > Original Estimate: 1h > Remaining Estimate: 1h > > We would like to add the ability to create and submit Spark jobs > programmatically via Scala/Java. We have found a way to hack this and submit > jobs via Yarn, but since > org.apache.spark.deploy.yarn.Client.main() > exits with either 0 or 1 in the end, this will mean exit of our own program. > We would like to add an optional spark conf param to NOT exit at the end of > the main -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
