Yeah I think the problem is that the spark-submit script doesn't pass the argument array to spark-class in the right way, so any quoted strings get flattened.
We do: ORIG_ARGS=$@ $SPARK_HOME/bin/spark-class org.apache.spark.deploy.SparkSubmit $ORIG_ARGS This works: // remove all the code relating to `shift`ing the arguments $SPARK_HOME/bin/spark-class org.apache.spark.deploy.SparkSubmit "$@" Not sure, but I think the issue is that when you make a copy of $@ in bash the type actually changes from an array to something else. My patch fixes this for spark-shell but I didn't realize that spark-submit does the same thing. https://github.com/apache/spark/pull/576/files#diff-bc287993dfd11fd18794041e169ffd72L23 I think we'll need to figure out how to do this correctly in the bash script so that quoted strings get passed in the right way. On Wed, Apr 30, 2014 at 1:06 PM, Marcelo Vanzin <van...@cloudera.com> wrote: > Just pulled again just in case. Verified your fix is there. > > $ ./bin/spark-submit --master yarn --deploy-mode client > --driver-java-options "-Dfoo -Dbar" blah blah blah > error: Unrecognized option '-Dbar'. > run with --help for more information or --verbose for debugging output > > > On Wed, Apr 30, 2014 at 12:49 PM, Patrick Wendell <pwend...@gmail.com> wrote: >> I added a fix for this recently and it didn't require adding -J >> notation - are you trying it with this patch? >> >> https://issues.apache.org/jira/browse/SPARK-1654 >> >> ./bin/spark-shell --driver-java-options "-Dfoo=a -Dbar=b" >> scala> sys.props.get("foo") >> res0: Option[String] = Some(a) >> scala> sys.props.get("bar") >> res1: Option[String] = Some(b) >> >> - Patrick >> >> On Wed, Apr 30, 2014 at 11:29 AM, Marcelo Vanzin <van...@cloudera.com> wrote: >>> Hello all, >>> >>> Maybe my brain is not evolved enough to be able to trace through what >>> happens with command-line arguments as they're parsed through all the >>> shell scripts... but I really can't figure out how to pass more than a >>> single JVM option on the command line. >>> >>> Unless someone has an obvious workaround that I'm missing, I'd like to >>> propose something that is actually pretty standard in JVM tools: using >>> -J. From javac: >>> >>> -J<flag> Pass <flag> directly to the runtime system >>> >>> So "javac -J-Xmx1g" would pass "-Xmx1g" to the underlying JVM. You can >>> use several of those to pass multiple options (unlike >>> --driver-java-options), so it helps that it's a short syntax. >>> >>> Unless someone has some issue with that I'll work on a patch for it... >>> (well, I'm going to do it locally for me anyway because I really can't >>> figure out how to do what I want to otherwise.) >>> >>> >>> -- >>> Marcelo > > > > -- > Marcelo
