[
https://issues.apache.org/jira/browse/SPARK-26190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16700896#comment-16700896
]
Marcelo Vanzin commented on SPARK-26190:
----------------------------------------
Just run your script without SparkLauncher? e.g. using
{{Runtime.getRuntime().exec()}}.
Or create a fake SPARK_HOME that is mostly symlinks to the original SPARK_HOME,
and has your custom spark-submit.
I don't think this is a good thing to have in Spark. It's a pretty obscure use
case and very easy for people to do the wrong thing and blame Spark.
> SparkLauncher: Allow users to set their own submitter script instead of
> hardcoded spark-submit
> ----------------------------------------------------------------------------------------------
>
> Key: SPARK-26190
> URL: https://issues.apache.org/jira/browse/SPARK-26190
> Project: Spark
> Issue Type: Improvement
> Components: Java API, Spark Core, Spark Submit
> Affects Versions: 2.1.0
> Environment: Apache Spark 2.0.1 on yarn cluster (MapR distribution)
> Reporter: Gyanendra Dwivedi
> Priority: Major
>
> Currently the script name is hard-coded in the 'createBuilder()' method of
> org.apache.spark.launcher.SparkLauncher class:
> {code:java}
> // code placeholder
> private ProcessBuilder createBuilder() {
> List<String> cmd = new ArrayList();
> String script = CommandBuilderUtils.isWindows() ? "spark-submit.cmd" :
> "spark-submit";
> cmd.add(CommandBuilderUtils.join(File.separator, new
> String[]{this.builder.getSparkHome(), "bin", script}));
> cmd.addAll(this.builder.buildSparkSubmitArgs());
> ......
> ......
> }{code}
>
>
> It has following issues, which prevents its usage in certain scenario.
> 1) Developer may not use their own custom scripts with different name. They
> are forced to use the one shipped with the installation. Overwriting that may
> not be the option, when it is not allowed to alter the original installation.
> 2) The code expect the script to be present at "SPARK_HOME/bin" folder.
> 3) The 'createBuilder()' method is private and hence, extending the
> 'org.apache.spark.launcher.SparkLauncher' is not an option.
>
> Proposed solution:
> 1) Developer should be given option to set their own custom script, which may
> be located at any path.
> 2) Only in case the parameter is not set, the default should be taken from
> SPARK_HOME/bin folder.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
