That's a great idea. I did what you suggested and added the url to the props file in the uri of the json. The properties file now shows up in the sandbox. But when it goes to run spark-submit with "--properties-file props.properties" it fails to find it:
Exception in thread "main" java.lang.IllegalArgumentException: requirement failed: Properties file props.properties does not exist On 11 June 2015 at 22:17, Matthew Jones <mle...@gmail.com> wrote: > If you are using chronos you can just put the url in the task json and > chronos will download it into your sandbox. Then just use spark-submit > --properties-file app.properties. > > On Thu, 11 Jun 2015 15:52 Marcelo Vanzin <van...@cloudera.com> wrote: > >> That's not supported. You could use wget / curl to download the file to a >> temp location before running spark-submit, though. >> >> On Thu, Jun 11, 2015 at 12:48 PM, Gary Ogden <gog...@gmail.com> wrote: >> >>> I have a properties file that is hosted at a url. I would like to be >>> able to use the url in the --properties-file parameter when submitting a >>> job to mesos using spark-submit via chronos >>> >>> I would rather do this than use a file on the local server. >>> >>> This doesn't seem to work though when submitting from chronos: >>> >>> bin/spark-submit --properties-file http://server01/props/app.properties >>> >>> >>> Inside the properties file: >>> spark.executor.memory=256M >>> spark.cores.max=1 >>> spark.shuffle.consolidateFiles=true >>> spark.task.cpus=1 >>> spark.deploy.defaultCores=1 >>> spark.driver.cores=1 >>> spark.scheduler.mode=FAIR >>> >>> So how do I specify a properties file in a url? >>> >> >> >> >> -- >> Marcelo >> >
