Could you build a fat jar by including all your dependencies along with you application. See here <http://spark.apache.org/docs/latest/submitting-applications.html#advanced-dependency-management> and here <http://spark.apache.org/docs/latest/submitting-applications.html#bundling-your-applications-dependencies> .
Also: **So this application-jar can point to a directory and will be expanded? Orneeds to be a path to a single specific jar?** *This will be path to a single specific JAR.* On Tue, Jan 12, 2016 at 12:04 PM, jiml <[email protected]> wrote: > Question is: Looking for all the ways to specify a set of jars using --jars > on spark-submit > > I know this is old but I am about to submit a proposed docs change on > --jars, and I had an issue with --jars today > > When this user submitted the following command line, is that a proper way > to > reference a jar? > > hdfs://master:8000/srcdata/kmeans (is that a directory? or a jar that > doesn't end with .jar? I have not gotten into the machine learning libs yet > to recognize this) > > I know the docs say, "Path to a bundled jar including your application and > all dependencies. The URL must be globally visible inside of your cluster, > for instance, an hdfs:// path or a file:// path that is present on all > nodes." > > *So this application-jar can point to a directory and will be expanded? Or > needs to be a path to a single specific jar?* > > I ask because when I was testing --jars today, we had to explicitly provide > a path to each jar: > > //usr/local/spark/bin/spark-submit --class jpsgcs.thold.PipeLinkageData > > ---jars=local:/usr/local/spark/jars/groovy-all-2.3.3.jar,local:/usr/local/spark/jars/guava-14.0.1.jar,local:/usr/local/spark/jars/jopt-simple-4.6.jar,local:/usr/local/spark/jars/jpsgcs-core-1.0.8-2.jar,local:/usr/local/spark/jars/jpsgcs-pipe-1.0.6-7.jar > /usr/local/spark/jars/thold-0.0.1-1.jar/ > > (The only way I figured out to use the commas was a StackOverflow answer > that led me to look beyond the docs to the command line: spark-submit > --help > results in : > > --jars JARS Comma-separated list of local jars to include > on the driver > and executor classpaths. > > > And it seems that we do not need to put the main jar in the --jars > argument, > I have not tested yet if other classes in the application-jar > (/usr/local/spark/jars/thold-0.0.1-1.jar) are shipped to workers, or if I > need to put the application-jar in the --jars path to get classes not named > after --class to be seen? > > Thanks for any ideas > > > > > -- > View this message in context: > http://apache-spark-user-list.1001560.n3.nabble.com/how-to-submit-multiple-jar-files-when-using-spark-submit-script-in-shell-tp16662p25942.html > Sent from the Apache Spark User List mailing list archive at Nabble.com. > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [email protected] > For additional commands, e-mail: [email protected] > >
