No, and it looks like a problem. 2.2. --master yarn --deploy-mode client means: 1. submit spark as yarn app, but spark-driver is started on local machine. 2. A upload all dependent jars to HDFS and specify jar HDFS paths in --jars arg. 3. Driver runs my Spark Application main class named "MySuperSparkJob" and MySuperSparkJob fails because it doesn't get jars, thay are all in HDFS and not accessible from local machine...
2016-05-17 15:18 GMT+02:00 Jeff Zhang <zjf...@gmail.com>: > Do you put your app jar on hdfs ? The app jar must be on your local > machine. > > On Tue, May 17, 2016 at 8:33 PM, Serega Sheypak <serega.shey...@gmail.com> > wrote: > >> hi, I'm trying to: >> 1. upload my app jar files to HDFS >> 2. run spark-submit with: >> 2.1. --master yarn --deploy-mode cluster >> or >> 2.2. --master yarn --deploy-mode client >> >> specifying --jars hdfs:///my/home/commons.jar,hdfs:///my/home/super.jar >> >> When spark job is submitted, SparkSubmit client outputs: >> Warning: Skip remote jar hdfs:///user/baba/lib/akka-slf4j_2.11-2.3.11.jar >> ... >> >> and then spark application main class fails with class not found >> exception. >> Is there any workaround? >> > > > > -- > Best Regards > > Jeff Zhang >
