Question is: Looking for all the ways to specify a set of jars using --jars
on spark-submit
I know this is old but I am about to submit a proposed docs change on
--jars, and I had an issue with --jars today
When this user submitted the following command line, is that a proper way to
reference a jar?
hdfs://master:8000/srcdata/kmeans (is that a directory? or a jar that
doesn't end with .jar? I have not gotten into the machine learning libs yet
to recognize this)
I know the docs say, "Path to a bundled jar including your application and
all dependencies. The URL must be globally visible inside of your cluster,
for instance, an hdfs:// path or a file:// path that is present on all
nodes."
*So this application-jar can point to a directory and will be expanded? Or
needs to be a path to a single specific jar?*
I ask because when I was testing --jars today, we had to explicitly provide
a path to each jar:
//usr/local/spark/bin/spark-submit --class jpsgcs.thold.PipeLinkageData
---jars=local:/usr/local/spark/jars/groovy-all-2.3.3.jar,local:/usr/local/spark/jars/guava-14.0.1.jar,local:/usr/local/spark/jars/jopt-simple-4.6.jar,local:/usr/local/spark/jars/jpsgcs-core-1.0.8-2.jar,local:/usr/local/spark/jars/jpsgcs-pipe-1.0.6-7.jar
/usr/local/spark/jars/thold-0.0.1-1.jar/
(The only way I figured out to use the commas was a StackOverflow answer
that led me to look beyond the docs to the command line: spark-submit --help
results in :
--jars JARS Comma-separated list of local jars to include
on the driver
and executor classpaths.
And it seems that we do not need to put the main jar in the --jars argument,
I have not tested yet if other classes in the application-jar
(/usr/local/spark/jars/thold-0.0.1-1.jar) are shipped to workers, or if I
need to put the application-jar in the --jars path to get classes not named
after --class to be seen?
Thanks for any ideas
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/how-to-submit-multiple-jar-files-when-using-spark-submit-script-in-shell-tp16662p25942.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
