Hmm either spark-submit isn't picking up the relative path or Chronos is not setting your working directory to your sandbox. Try using "cd $MESOS_SANDBOX && spark-submit --properties-file props.properties"
On Fri, Jun 12, 2015 at 12:32 PM Gary Ogden <gog...@gmail.com> wrote: > That's a great idea. I did what you suggested and added the url to the > props file in the uri of the json. The properties file now shows up in the > sandbox. But when it goes to run spark-submit with "--properties-file > props.properties" it fails to find it: > > Exception in thread "main" java.lang.IllegalArgumentException: requirement > failed: Properties file props.properties does not exist > > > On 11 June 2015 at 22:17, Matthew Jones <mle...@gmail.com> wrote: > >> If you are using chronos you can just put the url in the task json and >> chronos will download it into your sandbox. Then just use spark-submit >> --properties-file app.properties. >> >> On Thu, 11 Jun 2015 15:52 Marcelo Vanzin <van...@cloudera.com> wrote: >> >>> That's not supported. You could use wget / curl to download the file to >>> a temp location before running spark-submit, though. >>> >>> On Thu, Jun 11, 2015 at 12:48 PM, Gary Ogden <gog...@gmail.com> wrote: >>> >>>> I have a properties file that is hosted at a url. I would like to be >>>> able to use the url in the --properties-file parameter when submitting a >>>> job to mesos using spark-submit via chronos >>>> >>>> I would rather do this than use a file on the local server. >>>> >>>> This doesn't seem to work though when submitting from chronos: >>>> >>>> bin/spark-submit --properties-file http://server01/props/app.properties >>>> >>>> >>>> Inside the properties file: >>>> spark.executor.memory=256M >>>> spark.cores.max=1 >>>> spark.shuffle.consolidateFiles=true >>>> spark.task.cpus=1 >>>> spark.deploy.defaultCores=1 >>>> spark.driver.cores=1 >>>> spark.scheduler.mode=FAIR >>>> >>>> So how do I specify a properties file in a url? >>>> >>> >>> >>> >>> -- >>> Marcelo >>> >> >
