[
https://issues.apache.org/jira/browse/SPARK-26190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16700890#comment-16700890
]
Marcelo Vanzin commented on SPARK-26190:
----------------------------------------
Based solely on what you wrote here, I'm leaning towards closing this.
SparkLauncher is a programatic API around spark-submit, not around your custom
script. If you have a custom script, you can call it without using
SparkLauncher.
> SparkLauncher: Allow users to set their own submitter script instead of
> hardcoded spark-submit
> ----------------------------------------------------------------------------------------------
>
> Key: SPARK-26190
> URL: https://issues.apache.org/jira/browse/SPARK-26190
> Project: Spark
> Issue Type: Improvement
> Components: Java API, Spark Core, Spark Submit
> Affects Versions: 2.1.0
> Environment: Apache Spark 2.0.1 on yarn cluster (MapR distribution)
> Reporter: Gyanendra Dwivedi
> Priority: Major
>
> Currently the script name is hard-coded in the 'createBuilder()' method of
> org.apache.spark.launcher.SparkLauncher class:
>
> private ProcessBuilder createBuilder() {
> List<String> cmd = new ArrayList();
> String script = CommandBuilderUtils.isWindows() ? "*spark-submit.cmd*" :
> "*spark-submit*";
> cmd.add(CommandBuilderUtils.join(File.separator, new
> String[]\{this.builder.getSparkHome(), "bin", script}));
> cmd.addAll(this.builder.buildSparkSubmitArgs());
> ............
> .............
> }
>
> It has following issues, which prevents its usage in certain scenario.
> 1) Developer may not use their own custom scripts with different name. They
> are forced to use the one shipped with the installation. Overwriting that may
> not be the option, when it is not allowed to alter the original installation.
> 2) The code expect the script to be present at "SPARK_HOME/bin" folder.
> 3) The 'createBuilder()' method is private and hence, extending the
> 'org.apache.spark.launcher.SparkLauncher' is not an option.
>
> Proposed solution:
> 1) Developer should be given option to set their own custom script, which may
> be located at any path.
> 2) Only in case the parameter is not set, the default should be taken from
> SPARK_HOME/bin folder.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
