Hi, I know about that approach. I don't want to run mess of classes from single jar, I want to utilize distributed cache functionality and ship application jar and dependent jars explicitly. --deploy-mode client unfortunately copies and distributes all jars repeatedly for every spark job started from driver class...
2016-05-17 15:41 GMT+02:00 <spark....@yahoo.com>: > Hi Serega, > > Create a jar including all the the dependencies and execute it like below > through shell script > > /usr/local/spark/bin/spark-submit \ //location of your spark-submit > --class classname \ //location of your main classname > --master yarn \ > --deploy-mode cluster \ > /home/hadoop/SparkSampleProgram.jar //location of your jar file > > Thanks > Raj > > > > Sent from Yahoo Mail. Get the app <https://yho.com/148vdq> > > > On Tuesday, May 17, 2016 6:03 PM, Serega Sheypak <serega.shey...@gmail.com> > wrote: > > > hi, I'm trying to: > 1. upload my app jar files to HDFS > 2. run spark-submit with: > 2.1. --master yarn --deploy-mode cluster > or > 2.2. --master yarn --deploy-mode client > > specifying --jars hdfs:///my/home/commons.jar,hdfs:///my/home/super.jar > > When spark job is submitted, SparkSubmit client outputs: > Warning: Skip remote jar hdfs:///user/baba/lib/akka-slf4j_2.11-2.3.11.jar > ... > > and then spark application main class fails with class not found exception. > Is there any workaround? > > >
