[
https://issues.apache.org/jira/browse/SPARK-20016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15935005#comment-15935005
]
Marcelo Vanzin commented on SPARK-20016:
----------------------------------------
This was a long time ago and mostly trial & error, since Windows batch files
make no sense. Since I don't really have a Windows test env anymore, I'd
appreciated if someone who does have one can try things out.
> SparkLauncher submit job failed after setConf with special charaters under
> windows
> ----------------------------------------------------------------------------------
>
> Key: SPARK-20016
> URL: https://issues.apache.org/jira/browse/SPARK-20016
> Project: Spark
> Issue Type: Bug
> Components: Java API
> Affects Versions: 2.0.0
> Environment: windows 7, 8, 10, 2008, 2008R2, etc.
> Reporter: Vincent Sun
>
> I am using sparkLauncher JAVA API to submit job to a remote spark cluster
> master. Codes looks like follow:
> /*
> * launch Job
> */
> public static void launch() throws Exception {
> SparkLauncher spark = new SparkLauncher();
> spark.setAppName("sparkdemo").setAppResource("hdfs://10.250.1.121:9000/application.jar").setMainClass("test.Application");
> spark.setMaster(spark://10.250.1.120:6066);
> spark.setDeployMode("cluster");
> spark.setConf("spark.executor.cores","2")
> spark.setConf("spark.executor.memory","8G")
> spark.startApplication(new MyAppListener(job.getAppName()));
> }
> It works fine under Linux/CentOS, but failed on my own desktop which is a
> windows 8 OS. It will throw out error:
> <b>[launcher-proc-1] The filename, directory name, or volume label syntax is
> incorrect.</b>
> The finial command I caught is this:
> spark-submit.cmd --master spark://10.250.1.120:6066 --deploy-mode cluster
> --name sparkdemo --conf "spark.executor.memory=8G" --conf
> "spark.executor.cores=2" --class test.Application
> hdfs://10.250.1.121:9000/application.jar
> The quote on spark.executor.memory=8G and spark.executor.cores=2 cause the
> exception.
> After debug into the source code I found the reason is at:
> quoteForBatchScript method of CommandBuilderUtils class
> It will add quotes while there is '=' or some other kinds of special
> characters under windows system. Here is the source codes:
> static String quoteForBatchScript(String arg) {
> boolean needsQuotes = false;
> for (int i = 0; i < arg.length(); i++) {
> int c = arg.codePointAt(i);
> if (Character.isWhitespace(c) || c == '"' || c == '=' || c == ',' || c
> == ';') {
> needsQuotes = true;
> break;
> }
> }
> if (!needsQuotes) {
> return arg;
> }
> StringBuilder quoted = new StringBuilder();
> quoted.append("\"");
> for (int i = 0; i < arg.length(); i++) {
> int cp = arg.codePointAt(i);
> switch (cp) {
> case '"':
> quoted.append('"');
> break;
> default:
> break;
> }
> quoted.appendCodePoint(cp);
> }
> if (arg.codePointAt(arg.length() - 1) == '\\') {
> quoted.append("\\");
> }
> quoted.append("\"");
> return quoted.toString();
> }
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
