-- jars does indeed work but this causes the "jars" to also get shipped to the workers -- which I don't want to do for efficiency reasons.
I think you are saying that setting "spark.driver.extraClassPath" in spark-default.conf ought to have the same behavior as providing "--driver.class.apth" to spark-shell. Correct? If so I will file a bug report since this is definitely not the case. On Thu, Oct 30, 2014 at 5:39 PM, Matei Zaharia <matei.zaha...@gmail.com> wrote: > Try using --jars instead of the driver-only options; they should work with > spark-shell too but they may be less tested. > > Unfortunately, you do have to specify each JAR separately; you can maybe > use a shell script to list a directory and get a big list, or set up a > project that builds all of the dependencies into one assembly JAR. > > Matei > > > On Oct 30, 2014, at 5:24 PM, Shay Seng <s...@urbanengines.com> wrote: > > > > Hi, > > > > I've been trying to move up from spark 0.9.2 to 1.1.0. > > I'm getting a little confused with the setup for a few different use > cases, grateful for any pointers... > > > > (1) spark-shell + with jars that are only required by the driver > > (1a) > > I added "spark.driver.extraClassPath /mypath/to.jar" to my > spark-defaults.conf > > I launched spark-shell with: ./spark-shell > > > > Here I see on the WebUI that spark.driver.extraClassPath has been set, > but I am NOT able to access any methods in the jar. > > > > (1b) > > I removed "spark.driver.extraClassPath" from my spark-default.conf > > I launched spark-shell with .//spark-shell --driver.class.path > /mypath/to.jar > > > > Again I see that the WebUI spark.driver.extraClassPath has been set. > > But this time I am able to access the methods in the jar. > > > > Q: Is spark-shell not considered the driver in this case? why does > using --driver.class.path on the command line have a different behavior to > setting it in spark-defaults.conf ? > > > > > > (2) Rather than adding each jar individually, is there a way to use > wildcards? Previously with SPARK_CLASS_PATH I was able to use <mypath>/* > but with --driver.class.path it seems to require individual files. > > > > tks > > Shay > >
