On Fri, Jun 20, 2014 at 8:22 AM, Koert Kuipers <ko...@tresata.com> wrote: > thanks! i will try that. > i guess what i am most confused about is why the executors are trying to > retrieve the jars directly using the info i provided to add jars to my spark > context. i mean, thats bound to fail no? i could be on a different machine > (so my file://) isnt going to work for them, or i could have the jars in a > directory that is only readable by me. > > how come the jars are not just shipped to yarn as part of the job submittal?
They are if they are specified correctly. Check the guide: http://spark.apache.org/docs/latest/submitting-applications.html See the "Advanced Dependency Management" section. Your default filesystem is probably hdfs, which means that if you provide a path with no protocol, the executors will consider it as an hdfs path, and it won't work if you're pointing at a file that exists in your local fs. -- Marcelo
