Hi Guys,
Spark rookie here. I am getting a file not found exception on the --jars.
This is on the yarn cluster mode and I am running the following command on
our recently upgraded Spark 1.1.1 environment.
./bin/spark-submit --verbose --master yarn --deploy-mode cluster --class
myEngine --driver-memory 1g --driver-library-path
/hadoop/share/hadoop/mapreduce/lib/hadoop-lzo-0.4.18-201406111750.jar
--executor-memory 5g --executor-cores 5 --jars
/home/andy/spark/lib/joda-convert-1.2.jar --queue default --num-executors 4
/home/andy/spark/lib/my-spark-lib_1.0.jar
This is the error I am hitting. Any tips would be much appreciated. The
file permissions looks fine on my local disk.
14/10/13 22:49:39 INFO yarn.ApplicationMaster: Unregistering
ApplicationMaster with FAILED
14/10/13 22:49:39 INFO impl.AMRMClientImpl: Waiting for application to be
successfully unregistered.
Exception in thread "Driver" java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:162)
Caused by: org.apache.spark.SparkException: Job aborted due to stage
failure: Task 3 in stage 1.0 failed 4 times, most recent failure: Lost task
3.3 in stage 1.0 (TID 12, 122-67.vb2.company.com):
java.io.FileNotFoundException: ./joda-convert-1.2.jar (Permission denied)
java.io.FileOutputStream.open(Native Method)
java.io.FileOutputStream.<init>(FileOutputStream.java:221)
com.google.common.io.Files$FileByteSink.openStream(Files.java:223)
com.google.common.io.Files$FileByteSink.openStream(Files.java:211)
Thanks,
Andy
