Before running your script, could you confirm that " /data/software/spark-1.3.1-bin-2.4.0/applications/pss.am.core-1.0-SNAPSHOT-shaded.jar" exists? You might forget to build this jar.
Best Regards, Shixiong Zhu 2015-07-06 18:14 GMT+08:00 bit1...@163.com <bit1...@163.com>: > Hi， > I have following shell script that will submit the application to the > cluster. But whenever I start the application, I encounter > FileNotFoundException, after retrying for serveral times, I can > successfully submit it! > > > SPARK=/data/software/spark-1.3.1-bin-2.4.0 > APP_HOME=/data/software/spark-1.3.1-bin-2.4.0/applications > $SPARK/bin/spark-submit --deploy-mode cluster --name > PssAmStreamingApplication --master spark:/com-app1:7077 --driver-memory 1G > --executor-memory 4G --total-executor-cores 10 --class > com.app.PssAmStreamingApplicationDriver > $APP_HOME/pss.am.core-1.0-SNAPSHOT-shaded.jar > > > > > [root@com-app2 applications]# ./submitApplicationStreaming.sh > Running Spark using the REST application submission protocol. > Using Spark's default log4j profile: > org/apache/spark/log4j-defaults.properties > 15/07/06 18:05:35 INFO StandaloneRestClient: Submitting a request to > launch an application in spark://com-app1:7077. > Warning: Master endpoint spark://com-app1:7077 was not a REST server. > Falling back to legacy submission gateway instead. > 15/07/06 18:05:36 WARN NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > Sending launch command to spark://com-app1:7077 > Driver successfully submitted as driver-20150706180538-0008 > ... waiting before polling master for driver state > ... polling master for driver state > State of driver-20150706180538-0008 is ERROR > Exception from cluster was: java.io.FileNotFoundException: File > file:/data/software/spark-1.3.1-bin-2.4.0/applications/pss.am.core-1.0-SNAPSHOT-shaded.jar > does not exist > java.io.FileNotFoundException: File > file:/data/software/spark-1.3.1-bin-2.4.0/applications/pss.am.core-1.0-SNAPSHOT-shaded.jar > does not exist > at > org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:511) > > at > org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:724) > > at > org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:501) > > at > org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:397) > > at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:464) > at > org.apache.spark.deploy.worker.DriverRunner.org$apache$spark$deploy$worker$DriverRunner$$downloadUserJar(DriverRunner.scala:146) > > at > org.apache.spark.deploy.worker.DriverRunner$$anon$1.run(DriverRunner.scala:72) > > ------------------------------ > bit1...@163.com >
