[
https://issues.apache.org/jira/browse/SPARK-12176?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yuhang Chen updated SPARK-12176:
--------------------------------
Description:
The spark-submit uses '--conf K=V' pattern for setting configs. According to
the docs, if the 'V' you set has spaces in it, the whole 'K=V' parts should be
wrapped with quotes.
However, the SparkLauncher (org.apache.spark.launcher.SparkLauncher) would not
do that wrapping for you, and there is no chance for wrapping by yourself with
the API it provides.
For example, I want to add "-XX:+PrintGCDetails -XX:+PrintGCTimeStamps" for
executors (spark.executor.extraJavaOptions), and the conf contains a space in
it.
For spark-submit, I should wrap the conf with quotes like this:
--conf "spark.executor.extraJavaOptions=-XX:+PrintGCDetails
-XX:+PrintGCTimeStamps"
But when I use the setConf() API of SparkLauncher, I write code like this:
launcher.setConf("spark.executor.extraJavaOptions", "-XX:+PrintGCDetails
-XX:+PrintGCTimeStamps"). Now, SparkLauncher uses Java's ProcessBuilder to
start a sub-process, in which the spark-submit is finally executed. And it
turns out that the final command is like this;
--conf spark.executor.extraJavaOptions=-XX:+PrintGCDetails
-XX:+PrintGCTimeStamps
See? the quotes are gone, and the job counld not be launched with this command.
Then I checked up the source, all confs are stored in a Map before generating
launching commands. Thus. my advice is checking all values of the conf Map and
do wrapping during command building.
was:
The spark-submit uses '--conf K=V' pattern for setting configs. According to
the docs, if the 'V' you set has spaces in it, the whole 'K=V' parts should be
wrapped with quotes.
However, the SparkLauncher (org.apache.spark.launcher.SparkLauncher) would not
do that wrapping for you, and there is no chance for wrapping by yourself with
the API it provides.
I checked up the source, all confs are stored in a Map before generating
launching commands. Thus. my advice is checking all values of the conf Map and
do wrapping during command building.
For example, I want to add "-XX:+PrintGCDetails -XX:+PrintGCTimeStamps" for
executors (spark.executor.extraJavaOptions), and the conf contains a space in
it. For spark-submit, I should wrap the conf with quotes like this:
--conf "spark.executor.extraJavaOptions=-XX:+PrintGCDetails
-XX:+PrintGCTimeStamps"
But when I use the setConf() API of SparkLauncher, I write code like this:
launcher.setConf("spark.executor.extraJavaOptions", "-XX:+PrintGCDetails
-XX:+PrintGCTimeStamps"). Now, SparkLauncher uses Java's ProcessBuilder to
start a sub-process, in which the spark-submit is finally executed. I debugged
the source, and it turns out the command is like this;
--conf spark.executor.extraJavaOptions=-XX:+PrintGCDetails
-XX:+PrintGCTimeStamps
See? the quotes are gone, and the job counld not be launched with this command.
> SparkLauncher's setConf() does not support configs containing spaces
> --------------------------------------------------------------------
>
> Key: SPARK-12176
> URL: https://issues.apache.org/jira/browse/SPARK-12176
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
> Affects Versions: 1.4.0, 1.4.1, 1.5.0, 1.5.1, 1.5.2
> Environment: All
> Reporter: Yuhang Chen
> Priority: Minor
>
> The spark-submit uses '--conf K=V' pattern for setting configs. According to
> the docs, if the 'V' you set has spaces in it, the whole 'K=V' parts should
> be wrapped with quotes.
> However, the SparkLauncher (org.apache.spark.launcher.SparkLauncher) would
> not do that wrapping for you, and there is no chance for wrapping by yourself
> with the API it provides.
> For example, I want to add "-XX:+PrintGCDetails -XX:+PrintGCTimeStamps" for
> executors (spark.executor.extraJavaOptions), and the conf contains a space in
> it.
> For spark-submit, I should wrap the conf with quotes like this:
> --conf "spark.executor.extraJavaOptions=-XX:+PrintGCDetails
> -XX:+PrintGCTimeStamps"
> But when I use the setConf() API of SparkLauncher, I write code like this:
> launcher.setConf("spark.executor.extraJavaOptions", "-XX:+PrintGCDetails
> -XX:+PrintGCTimeStamps"). Now, SparkLauncher uses Java's ProcessBuilder to
> start a sub-process, in which the spark-submit is finally executed. And it
> turns out that the final command is like this;
> --conf spark.executor.extraJavaOptions=-XX:+PrintGCDetails
> -XX:+PrintGCTimeStamps
> See? the quotes are gone, and the job counld not be launched with this
> command.
> Then I checked up the source, all confs are stored in a Map before generating
> launching commands. Thus. my advice is checking all values of the conf Map
> and do wrapping during command building.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
