https://issues.apache.org/jira/browse/SPARK-10643
Looks like it's the reason... 2016-05-17 15:31 GMT+02:00 Serega Sheypak <serega.shey...@gmail.com>: > No, and it looks like a problem. > > 2.2. --master yarn --deploy-mode client > means: > 1. submit spark as yarn app, but spark-driver is started on local machine. > 2. A upload all dependent jars to HDFS and specify jar HDFS paths in > --jars arg. > 3. Driver runs my Spark Application main class named "MySuperSparkJob" and > MySuperSparkJob > fails because it doesn't get jars, thay are all in HDFS and not accessible > from local machine... > > > 2016-05-17 15:18 GMT+02:00 Jeff Zhang <zjf...@gmail.com>: > >> Do you put your app jar on hdfs ? The app jar must be on your local >> machine. >> >> On Tue, May 17, 2016 at 8:33 PM, Serega Sheypak <serega.shey...@gmail.com >> > wrote: >> >>> hi, I'm trying to: >>> 1. upload my app jar files to HDFS >>> 2. run spark-submit with: >>> 2.1. --master yarn --deploy-mode cluster >>> or >>> 2.2. --master yarn --deploy-mode client >>> >>> specifying --jars hdfs:///my/home/commons.jar,hdfs:///my/home/super.jar >>> >>> When spark job is submitted, SparkSubmit client outputs: >>> Warning: Skip remote jar >>> hdfs:///user/baba/lib/akka-slf4j_2.11-2.3.11.jar ... >>> >>> and then spark application main class fails with class not found >>> exception. >>> Is there any workaround? >>> >> >> >> >> -- >> Best Regards >> >> Jeff Zhang >> > >
