You can try adding that jar in SPARK_CLASSPATH (its deprecated though) in spark-env.sh file.
Thanks Best Regards On Tue, Jul 21, 2015 at 7:34 PM, Michal Haris <michal.ha...@visualdna.com> wrote: > I have a spark program that uses dataframes to query hive and I run it > both as a spark-shell for exploration and I have a runner class that > executes some tasks with spark-submit. I used to run against > 1.4.0-SNAPSHOT. Since then 1.4.0 and 1.4.1 were released so I tried to > switch to the official release. Now, when I run the program as a shell, > everything works but when I try to run it with spark-submit it complains > with this error: > > Exception in thread "main" java.lang.ClassNotFoundException: > java.lang.NoClassDefFoundError: > org/apache/hadoop/hive/ql/session/SessionState when creating Hive client > using classpath: file:/home/mharis/dxp-spark.jar > Please make sure that jars for your version of hive and hadoop are > included in the paths passed to spark.sql.hive.metastore.jars. > > What is suspicious is firstly 'using classpath: ...' where the jar is my > program, i.e. the paths that are passed along with --driver-class-path > option are missing. When I switch to an older 1.4.0-SNAPSHOT on the driver, > everything works. I observe the issue with 1.4.1. > > Are there any known obvious changes to how spark-submit handles > configuration that I have missed ? > > -- > Michal Haris > Technical Architect > direct line: +44 (0) 207 749 0229 > www.visualdna.com | t: +44 (0) 207 734 7033 > 31 Old Nichol Street > London > E2 7HR >
