Dean - our e-mails crossed, but thanks for the tip. Was independently arriving at your solution :)
Okay I'll submit something. - Patrick On Wed, Apr 30, 2014 at 2:14 PM, Marcelo Vanzin <van...@cloudera.com> wrote: > Cool, that seems to work. Thanks! > > On Wed, Apr 30, 2014 at 2:09 PM, Patrick Wendell <pwend...@gmail.com> wrote: >> Marcelo - Mind trying the following diff locally? If it works I can >> send a patch: >> >> patrick@patrick-t430s:~/Documents/spark$ git diff bin/spark-submit >> diff --git a/bin/spark-submit b/bin/spark-submit >> index dd0d95d..49bc262 100755 >> --- a/bin/spark-submit >> +++ b/bin/spark-submit >> @@ -18,7 +18,7 @@ >> # >> >> export SPARK_HOME="$(cd `dirname $0`/..; pwd)" >> -ORIG_ARGS=$@ >> +ORIG_ARGS=("$@") >> >> while (($#)); do >> if [ "$1" = "--deploy-mode" ]; then >> @@ -39,5 +39,5 @@ if [ ! -z $DRIVER_MEMORY ] && [ ! -z $DEPLOY_MODE ] >> && [ $DEPLOY_MODE = "client" >> export SPARK_MEM=$DRIVER_MEMORY >> fi >> >> -$SPARK_HOME/bin/spark-class org.apache.spark.deploy.SparkSubmit $ORIG_ARGS >> +$SPARK_HOME/bin/spark-class org.apache.spark.deploy.SparkSubmit >> "${ORIG_ARGS[@]}" >> >> On Wed, Apr 30, 2014 at 1:51 PM, Patrick Wendell <pwend...@gmail.com> wrote: >>> So I reproduced the problem here: >>> >>> == test.sh == >>> #!/bin/bash >>> for x in "$@"; do >>> echo "arg: $x" >>> done >>> ARGS_COPY=$@ >>> for x in "$ARGS_COPY"; do >>> echo "arg_copy: $x" >>> done >>> == >>> >>> ./test.sh a b "c d e" f >>> arg: a >>> arg: b >>> arg: c d e >>> arg: f >>> arg_copy: a b c d e f >>> >>> I'll dig around a bit more and see if we can fix it. Pretty sure we >>> aren't passing these argument arrays around correctly in bash. >>> >>> On Wed, Apr 30, 2014 at 1:48 PM, Marcelo Vanzin <van...@cloudera.com> wrote: >>>> On Wed, Apr 30, 2014 at 1:41 PM, Patrick Wendell <pwend...@gmail.com> >>>> wrote: >>>>> Yeah I think the problem is that the spark-submit script doesn't pass >>>>> the argument array to spark-class in the right way, so any quoted >>>>> strings get flattened. >>>>> >>>>> I think we'll need to figure out how to do this correctly in the bash >>>>> script so that quoted strings get passed in the right way. >>>> >>>> I tried a few different approaches but finally ended up giving up; my >>>> bash-fu is apparently not strong enough. If you can make it work >>>> great, but I have "-J" working locally in case you give up like me. >>>> :-) >>>> >>>> -- >>>> Marcelo > > > > -- > Marcelo