The thing is that the class it is complaining about is part of the spark
assembly jar, not in my extra jar. The assembly jar was compiled with
-Phive which is proven by the fact that it works with the same SPARK_HOME
when run as shell.
On 23 July 2015 at 17:33, Akhil Das wrote:
> You can try adding that jar in SPARK_CLASSPATH (its deprecated though) in
> spark-env.sh file.
>
> Thanks
> Best Regards
>
> On Tue, Jul 21, 2015 at 7:34 PM, Michal Haris
> wrote:
>
>> I have a spark program that uses dataframes to query hive and I run it
>> both as a spark-shell for exploration and I have a runner class that
>> executes some tasks with spark-submit. I used to run against
>> 1.4.0-SNAPSHOT. Since then 1.4.0 and 1.4.1 were released so I tried to
>> switch to the official release. Now, when I run the program as a shell,
>> everything works but when I try to run it with spark-submit it complains
>> with this error:
>>
>> Exception in thread "main" java.lang.ClassNotFoundException:
>> java.lang.NoClassDefFoundError:
>> org/apache/hadoop/hive/ql/session/SessionState when creating Hive client
>> using classpath: file:/home/mharis/dxp-spark.jar
>> Please make sure that jars for your version of hive and hadoop are
>> included in the paths passed to spark.sql.hive.metastore.jars.
>>
>> What is suspicious is firstly 'using classpath: ...' where the jar is my
>> program, i.e. the paths that are passed along with --driver-class-path
>> option are missing. When I switch to an older 1.4.0-SNAPSHOT on the driver,
>> everything works. I observe the issue with 1.4.1.
>>
>> Are there any known obvious changes to how spark-submit handles
>> configuration that I have missed ?
>>
>> --
>> Michal Haris
>> Technical Architect
>> direct line: +44 (0) 207 749 0229
>> www.visualdna.com | t: +44 (0) 207 734 7033
>> 31 Old Nichol Street
>> London
>> E2 7HR
>>
>
>
--
Michal Haris
Technical Architect
direct line: +44 (0) 207 749 0229
www.visualdna.com | t: +44 (0) 207 734 7033
31 Old Nichol Street
London
E2 7HR
