Unsubscribe

[LisTree Team]

LisTree Team - Big Data Training Team


-
To unsubscribe e-mail: user-unsubscr...@spark.apache.org

Unsubscribe

[LisTree Team]

LisTree Team - Big Data Training Team

-
To unsubscribe e-mail: user-unsubscr...@spark.apache.org

RE: spark-submit: file not found exception occurs

[LisTree Team]

you may use hdfs file not local file under yarn.


 Original Message 
Subject: spark-submit: file not found exception occurs
From: Shupeng Geng 
Date: Thu, June 15, 2017 8:14 pm
To: "user@spark.apache.org" ,
"d...@spark.apache.org" 

Hi, everyone,  An annoying problem occurs to me.   When submitting a spark job, the jar file not found exception is thrown as follows:    does not existread "main" java.io.FileNotFoundException: File file:/home/algo/shupeng/eeop_bridger/EeopBridger-1.0-SNAPSHOT.jar  at org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:534)  at org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:747)  at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:524)  at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:409)  at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:337)  at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:289)  at org.apache.spark.deploy.yarn.Client.copyFileToRemote(Client.scala:340)  at org.apache.spark.deploy.yarn.Client.org$apache$spark$deploy$yarn$Client$$distribute$1(Client.scala:433)  at org.apache.spark.deploy.yarn.Client$$anonfun$prepareLocalResources$10.apply(Client.scala:530)   at org.apache.spark.deploy.yarn.Client$$anonfun$prepareLocalResources$10.apply(Client.scala:529)    I’m pretty sure the file is located just where it should be.    Here is the submit shell cmd:    /usr/local/hadoop/spark-release/bin/spark-submit --master yarn --deploy-mode cluster --queue root.algo --name eeop_bridiger --class Bridger EeopBridger-1.0-SNAPSHOT.jar    Somebody give me some help, please. Many Thanks.   本邮件(包括任何附件）内容机密并受法律保护。如果您意外地收到这封邮件，请回复通知发件人并从当前系统中删除本邮件。任何未经授权者，严禁使用并部分或者全部的转发本条信息。任何未经授权的使用或传播都是被严格禁止的。远景能源与其分公司不对因不正确和不完整的转发此邮件包含的信息以及因任何因邮件延迟或对你的系统造成的损害而负责。远景能源不能保证此邮件的真实完整性,也不能确定此邮件是否含有病毒或者监听程序。  This email message (including any attachments) is confidential and may be legally privileged. If you have received it by mistake, please notify the sender by return email and delete this message from your system. Any unauthorized use or dissemination of this message in whole or in part is strictly prohibited. Envision Energy Limited and all its subsidiaries shall not be liable for the improper or incomplete transmission of the information contained in this email nor for any delay in its receipt or damage to your system. Envision Energy Limited does not guarantee the integrity of this email message, nor that this email message is free of viruses, interceptions, or interference.   



-
To unsubscribe e-mail: user-unsubscr...@spark.apache.org