Turns out one of the other developers wrapped the jobs in script and did a
cd to another folder in the script before executing spark-submit.
On 12 June 2015 at 14:20, Matthew Jones wrote:
> Hmm either spark-submit isn't picking up the relative path or Chronos is
> not setting your working direct
Hmm either spark-submit isn't picking up the relative path or Chronos is
not setting your working directory to your sandbox. Try using "cd
$MESOS_SANDBOX && spark-submit --properties-file props.properties"
On Fri, Jun 12, 2015 at 12:32 PM Gary Ogden wrote:
> That's a great idea. I did what you s
That's a great idea. I did what you suggested and added the url to the
props file in the uri of the json. The properties file now shows up in the
sandbox. But when it goes to run spark-submit with "--properties-file
props.properties" it fails to find it:
Exception in thread "main" java.lang.Il
If you are using chronos you can just put the url in the task json and
chronos will download it into your sandbox. Then just use spark-submit
--properties-file app.properties.
On Thu, 11 Jun 2015 15:52 Marcelo Vanzin wrote:
> That's not supported. You could use wget / curl to download the file t
That's not supported. You could use wget / curl to download the file to a
temp location before running spark-submit, though.
On Thu, Jun 11, 2015 at 12:48 PM, Gary Ogden wrote:
> I have a properties file that is hosted at a url. I would like to be able
> to use the url in the --properties-file p
I have a properties file that is hosted at a url. I would like to be able
to use the url in the --properties-file parameter when submitting a job to
mesos using spark-submit via chronos
I would rather do this than use a file on the local server.
This doesn't seem to work though when submitting fr