Hi Geoff - Appreciate the help here - I do understand what you’re saying below. And I am able to get this working when I submit a job to a local cluster.
I think part of the issue here is that there’s ambiguity in the terminology. When I say “LOCAL” spark, I mean an instance of spark that is created by my driver program, and is not a cluster itself. It means that my master node is “local”, and this mode is primarily used for testing. https://jaceklaskowski.gitbooks.io/mastering-apache-spark/content/spark-local.html <https://jaceklaskowski.gitbooks.io/mastering-apache-spark/content/spark-local.html> While I am able to get alluxio working with spark-submit, I am unable to get it working when using local mode. The mechanisms for setting class paths during spark-submit are not available in local mode. My understanding is that all one is able to use is: spark.conf.set(“”) To set any runtime properties of the local instance. Note that it is possible (and I am more convinced of this as time goes on) that alluxio simply does not work in spark local mode as described above. > On Apr 13, 2018, at 11:09 AM, Geoff Von Allmen <ge...@ibleducation.com> wrote: > > I fought with a ClassNotFoundException for quite some time, but it was for > kafka. > > The final configuration that got everything working was running spark-submit > with the following options: > > --jars "/path/to/.ivy2/jars/package.jar" \ > --driver-class-path "/path/to/.ivy2/jars/package.jar" \ > --conf "spark.executor.extraClassPath=/path/to/.ivy2/package.jar" \ > --packages org.some.package:package_name:version > While this was needed for me to run in cluster mode, it works equally well > for client mode as well. > > One other note when needing to supplied multiple items to these args - --jars > and --packages should be comma separated, --driver-class-path and > extraClassPath should be : separated > > HTH > > > On Fri, Apr 13, 2018 at 4:28 AM, jb44 <jbo...@gmail.com > <mailto:jbo...@gmail.com>> wrote: > Haoyuan - > > As I mentioned below, I've been through the documentation already. It has > not helped me to resolve the issue. > > Here is what I have tried so far: > > - setting extraClassPath as explained below > - adding fs.alluxio.impl through sparkconf > - adding spark.sql.hive.metastore.sharedPrefixes (though I don't believe > this matters in my case) > - compiling the client from source > > Do you have any other suggestions on how to get this working? > > Thanks > > > > -- > Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/ > <http://apache-spark-user-list.1001560.n3.nabble.com/> > > --------------------------------------------------------------------- > To unsubscribe e-mail: user-unsubscr...@spark.apache.org > <mailto:user-unsubscr...@spark.apache.org> > >
