[ https://issues.apache.org/jira/browse/SPARK-26190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16700896#comment-16700896 ]
Marcelo Vanzin commented on SPARK-26190: ---------------------------------------- Just run your script without SparkLauncher? e.g. using {{Runtime.getRuntime().exec()}}. Or create a fake SPARK_HOME that is mostly symlinks to the original SPARK_HOME, and has your custom spark-submit. I don't think this is a good thing to have in Spark. It's a pretty obscure use case and very easy for people to do the wrong thing and blame Spark. > SparkLauncher: Allow users to set their own submitter script instead of > hardcoded spark-submit > ---------------------------------------------------------------------------------------------- > > Key: SPARK-26190 > URL: https://issues.apache.org/jira/browse/SPARK-26190 > Project: Spark > Issue Type: Improvement > Components: Java API, Spark Core, Spark Submit > Affects Versions: 2.1.0 > Environment: Apache Spark 2.0.1 on yarn cluster (MapR distribution) > Reporter: Gyanendra Dwivedi > Priority: Major > > Currently the script name is hard-coded in the 'createBuilder()' method of > org.apache.spark.launcher.SparkLauncher class: > {code:java} > // code placeholder > private ProcessBuilder createBuilder() { > List<String> cmd = new ArrayList(); > String script = CommandBuilderUtils.isWindows() ? "spark-submit.cmd" : > "spark-submit"; > cmd.add(CommandBuilderUtils.join(File.separator, new > String[]{this.builder.getSparkHome(), "bin", script})); > cmd.addAll(this.builder.buildSparkSubmitArgs()); > ...... > ...... > }{code} > > > It has following issues, which prevents its usage in certain scenario. > 1) Developer may not use their own custom scripts with different name. They > are forced to use the one shipped with the installation. Overwriting that may > not be the option, when it is not allowed to alter the original installation. > 2) The code expect the script to be present at "SPARK_HOME/bin" folder. > 3) The 'createBuilder()' method is private and hence, extending the > 'org.apache.spark.launcher.SparkLauncher' is not an option. > > Proposed solution: > 1) Developer should be given option to set their own custom script, which may > be located at any path. > 2) Only in case the parameter is not set, the default should be taken from > SPARK_HOME/bin folder. -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org