Re: spark-submit not using conf/spark-defaults.conf

I think it's a missing feature. On Wed, Sep 2, 2015 at 10:58 PM, Axel Dahl wrote: > So a bit more investigation, shows that: > > if I have configured spark-defaults.conf with: > > "spark.files library.py" > > then if I call > > "spark-submit.py -v test.py" > > I

Re: spark-submit not using conf/spark-defaults.conf

logged it here: https://issues.apache.org/jira/browse/SPARK-10436 On Thu, Sep 3, 2015 at 10:32 AM, Davies Liu wrote: > I think it's a missing feature. > > On Wed, Sep 2, 2015 at 10:58 PM, Axel Dahl wrote: > > So a bit more investigation, shows

spark-submit not using conf/spark-defaults.conf

in my spark-defaults.conf I have: spark.files file1.zip, file2.py spark.master spark://master.domain.com:7077 If I execute: bin/pyspark I can see it adding the files correctly. However if I execute bin/spark-submit test.py where test.py relies on the file1.zip, I get

Re: spark-submit not using conf/spark-defaults.conf

This should be a bug, could you create a JIRA for it? On Wed, Sep 2, 2015 at 4:38 PM, Axel Dahl wrote: > in my spark-defaults.conf I have: > spark.files file1.zip, file2.py > spark.master spark://master.domain.com:7077 > > If I execute: >

Re: spark-submit not using conf/spark-defaults.conf

So a bit more investigation, shows that: if I have configured spark-defaults.conf with: "spark.files library.py" then if I call "spark-submit.py -v test.py" I see that my "spark.files" default option has been replaced with "spark.files test.py", basically spark-submit is