You can look into spark.driver.userClassPathFirst flag. spark.driver.userClassPathFirstfalse(Experimental) Whether to give user-added jars precedence over Spark's own jars when loading classes in the the driver. This feature can be used to mitigate conflicts between Spark's dependencies and user dependencies. It is currently an experimental feature. This is used in cluster mode only.
Thanks Best Regards On Thu, Jun 25, 2015 at 1:08 AM, Yana Kadiyska <yana.kadiy...@gmail.com> wrote: > Thanks, that did seem to make a difference. I am a bit scared of this > approach as spark itself has a different guava dependency but the error > does go away this way > > On Wed, Jun 24, 2015 at 10:04 AM, Akhil Das <ak...@sigmoidanalytics.com> > wrote: > >> Can you try to add those jars in the SPARK_CLASSPATH and give it a try? >> >> Thanks >> Best Regards >> >> On Wed, Jun 24, 2015 at 12:07 AM, Yana Kadiyska <yana.kadiy...@gmail.com> >> wrote: >> >>> Hi folks, I have been using Spark against an external Metastore service >>> which runs Hive with Cdh 4.6 >>> >>> In Spark 1.2, I was able to successfully connect by building with the >>> following: >>> >>> ./make-distribution.sh --tgz -Dhadoop.version=2.0.0-mr1-cdh4.2.0 >>> -Phive-thriftserver -Phive-0.12.0 >>> >>> I see that in Spark 1.4 the Hive 0.12.0 profile is deprecated in favor >>> of >>> spark.sql.hive.metastore.version/spark.sql.hive.metastore.jars >>> >>> When I tried to use this setup spark-shell fails for me with the >>> following error: >>> >>> 15/06/23 18:18:07 INFO hive.HiveContext: Initializing >>> HiveMetastoreConnection version 0.12.0 using [Ljava.net.URL;@7b7a9a6c >>> java.lang.ClassNotFoundException: java.lang.NoClassDefFoundError: >>> com/google/common/base/Preconditions when creating Hive client using >>> classpath: file:/hive/lib/guava-11.0.2.jar, >>> file:/hive/lib/hive-exec-0.10.0-cdh4.6.0.jar, >>> file:/hive/lib/hive-metastore-0.10.0-cdh4.6.0.jar, >>> file:/hadoop/share/hadoop/mapreduce1/lib/hadoop-common-2.0.0-cdh4.6.0.jar, >>> file:/hive/lib/commons-logging-1.0.4.jar >>> >>> >>> >>> I don't know why it's not seeing the class -- it's in the guava jar. If >>> anyone has had success with 0.12 version please let me know what jars need >>> to be on the classpath. I think my Hive version might be too outdated but I >>> don't control the metastore and I had success with Spark1.2 so I'm hoping... >>> >>> >>> >> >
