Or if it has something to do with the way you package your files - try another alternative method and see if it works
On Monday, October 13, 2014, HARIPRIYA AYYALASOMAYAJULA < aharipriy...@gmail.com> wrote: > Well in the cluster, can you try copying the entire folder and then run? > For example my home folder say helloWorld consists of the src, target etc. > can you copy the entire folder in the cluster ? I doubt it is looking for > some dependencies and is missing that when it runs your jar file. > or if you mention some input file which your program uses, ensure that it > is in hdfs and correct path is specified. > > These were a few things I checked when I got a filenotfound exception > > On Monday, October 13, 2014, Jimmy McErlain <ji...@sellpoints.com > <javascript:_e(%7B%7D,'cvml','ji...@sellpoints.com');>> wrote: > >> That didnt seem to work... the jar files are in the target > scala2.10 >> folder when I package, then I move the jar to the cluster and launch the >> app... still the same error... Thoughts? >> J >> ᐧ >> >> >> >> >> *JIMMY MCERLAIN* >> >> DATA SCIENTIST (NERD) >> >> *. . . . . . . . . . . . . . . . . .* >> >> >> *IF WE CAN’T DOUBLE YOUR SALES,* >> >> >> >> *ONE OF US IS IN THE WRONG BUSINESS.* >> >> *E*: ji...@sellpoints.com >> >> *M*: *510.303.7751* >> >> On Mon, Oct 13, 2014 at 5:39 PM, HARIPRIYA AYYALASOMAYAJULA < >> aharipriy...@gmail.com> wrote: >> >>> Helo, >>> >>> Can you check if the jar file is available in the target->scala-2.10 >>> folder? >>> >>> When you use sbt package to make the jar file, that is where the jar >>> file would be located. >>> >>> The following command works well for me: >>> >>> spark-submit --class “Classname" --master yarn-cluster >>> jarfile(withcomplete path) >>> >>> Can you try checking with this initially and later add other options? >>> >>> On Mon, Oct 13, 2014 at 7:36 PM, Jimmy <ji...@sellpoints.com> wrote: >>> >>>> Having the exact same error with the exact same jar.... Do you work for >>>> Altiscale? :) >>>> J >>>> >>>> Sent from my iPhone >>>> >>>> On Oct 13, 2014, at 5:33 PM, Andy Srine <andy.sr...@gmail.com> wrote: >>>> >>>> Hi Guys, >>>> >>>> >>>> Spark rookie here. I am getting a file not found exception on the >>>> --jars. This is on the yarn cluster mode and I am running the following >>>> command on our recently upgraded Spark 1.1.1 environment. >>>> >>>> >>>> ./bin/spark-submit --verbose --master yarn --deploy-mode cluster >>>> --class myEngine --driver-memory 1g --driver-library-path >>>> /hadoop/share/hadoop/mapreduce/lib/hadoop-lzo-0.4.18-201406111750.jar >>>> --executor-memory 5g --executor-cores 5 --jars >>>> /home/andy/spark/lib/joda-convert-1.2.jar --queue default --num-executors 4 >>>> /home/andy/spark/lib/my-spark-lib_1.0.jar >>>> >>>> >>>> This is the error I am hitting. Any tips would be much appreciated. The >>>> file permissions looks fine on my local disk. >>>> >>>> >>>> 14/10/13 22:49:39 INFO yarn.ApplicationMaster: Unregistering >>>> ApplicationMaster with FAILED >>>> >>>> 14/10/13 22:49:39 INFO impl.AMRMClientImpl: Waiting for application to >>>> be successfully unregistered. >>>> >>>> Exception in thread "Driver" java.lang.reflect.InvocationTargetException >>>> >>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >>>> >>>> at >>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) >>>> >>>> at >>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >>>> >>>> at java.lang.reflect.Method.invoke(Method.java:606) >>>> >>>> at >>>> org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:162) >>>> >>>> Caused by: org.apache.spark.SparkException: Job aborted due to stage >>>> failure: Task 3 in stage 1.0 failed 4 times, most recent failure: Lost task >>>> 3.3 in stage 1.0 (TID 12, 122-67.vb2.company.com): >>>> java.io.FileNotFoundException: ./joda-convert-1.2.jar (Permission denied) >>>> >>>> java.io.FileOutputStream.open(Native Method) >>>> >>>> java.io.FileOutputStream.<init>(FileOutputStream.java:221) >>>> >>>> >>>> com.google.common.io.Files$FileByteSink.openStream(Files.java:223) >>>> >>>> >>>> com.google.common.io.Files$FileByteSink.openStream(Files.java:211) >>>> >>>> >>>> Thanks, >>>> Andy >>>> >>>> >>> >>> >>> -- >>> Regards, >>> Haripriya Ayyalasomayajula >>> >>> >> > > -- > Regards, > Haripriya Ayyalasomayajula > contact : 650-796-7112 > > -- Regards, Haripriya Ayyalasomayajula contact : 650-796-7112
