+ if exists whether it has read permission for the user who tries to run the job.
Regards Vivek On Tue, Dec 29, 2015 at 6:56 am, Ted Yu <yuzhih...@gmail.com<mailto:yuzhih...@gmail.com>> wrote: Have you verified that the following file does exist ? /home/hadoop/git/scalaspark/./target/scala-2.10/cluster-incidents_2.10-1.0.jar Thanks On Mon, Dec 28, 2015 at 3:16 PM, Daniel Valdivia <h...@danielvaldivia.com<mailto:h...@danielvaldivia.com>> wrote: Hi, I'm trying to submit a job to a small spark cluster running in stand alone mode, however it seems like the jar file I'm submitting to the cluster is "not found" by the workers nodes. I might have understood wrong, but I though the Driver node would send this jar file to the worker nodes, or should I manually send this file to each worker node before I submit the job? what I'm doing: $SPARK_HOME/bin/spark-submit --master spark://sslabnode01:6066 --deploy-mode cluster --class ClusterIncidents ./target/scala-2.10/cluster-incidents_2.10-1.0.jar The error I'm getting: Running Spark using the REST application submission protocol. Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties 15/12/28 15:13:58 INFO RestSubmissionClient: Submitting a request to launch an application in spark://sslabnode01:6066. 15/12/28 15:13:59 INFO RestSubmissionClient: Submission successfully created as driver-20151228151359-0003. Polling submission state... 15/12/28 15:13:59 INFO RestSubmissionClient: Submitting a request for the status of submission driver-20151228151359-0003 in spark://sslabnode01:6066. 15/12/28 15:13:59 INFO RestSubmissionClient: State of driver driver-20151228151359-0003 is now ERROR. 15/12/28 15:13:59 INFO RestSubmissionClient: Driver is running on worker worker-20151218150246-10.15.235.241-52077 at 10.15.235.241:52077<http://10.15.235.241:52077>. 15/12/28 15:13:59 ERROR RestSubmissionClient: Exception from the cluster: java.io.FileNotFoundException: /home/hadoop/git/scalaspark/./target/scala-2.10/cluster-incidents_2.10-1.0.jar (No such file or directory) java.io.FileInputStream.open(Native Method) java.io.FileInputStream.<init>(FileInputStream.java:146) org.spark-project.guava.io.Files$FileByteSource.openStream(Files.java:124) org.spark-project.guava.io.Files$FileByteSource.openStream(Files.java:114) org.spark-project.guava.io.ByteSource.copyTo(ByteSource.java:202) org.spark-project.guava.io.Files.copy(Files.java:436) org.apache.spark.util.Utils$.org$apache$spark$util$Utils$$copyRecursive(Utils.scala:514) org.apache.spark.util.Utils$.copyFile(Utils.scala:485) org.apache.spark.util.Utils$.doFetchFile(Utils.scala:562) org.apache.spark.util.Utils$.fetchFile(Utils.scala:369) org.apache.spark.deploy.worker.DriverRunner.org<http://org.apache.spark.deploy.worker.DriverRunner.org>$apache$spark$deploy$worker$DriverRunner$$downloadUserJar(DriverRunner.scala:150) org.apache.spark.deploy.worker.DriverRunner$$anon$1.run(DriverRunner.scala:79) 15/12/28 15:13:59 INFO RestSubmissionClient: Server responded with CreateSubmissionResponse: { "action" : "CreateSubmissionResponse", "message" : "Driver successfully submitted as driver-20151228151359-0003", "serverSparkVersion" : "1.5.2", "submissionId" : "driver-20151228151359-0003", "success" : true } Thanks in advance --------------------------------------------------------------------- To unsubscribe, e-mail: user-unsubscr...@spark.apache.org<mailto:user-unsubscr...@spark.apache.org> For additional commands, e-mail: user-h...@spark.apache.org<mailto:user-h...@spark.apache.org> The information contained in this electronic message and any attachments to this message are intended for the exclusive use of the addressee(s) and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you should not disseminate, distribute or copy this e-mail. Please notify the sender immediately and destroy all copies of this message and any attachments. WARNING: Computer viruses can be transmitted via email. The recipient should check this email and any attachments for the presence of viruses. The company accepts no liability for any damage caused by any virus transmitted by this email. www.wipro.com
