[
https://issues.apache.org/jira/browse/SPARK-11702?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15005074#comment-15005074
]
Marcelo Vanzin commented on SPARK-11702:
----------------------------------------
Sean, it's a real bug. He's not trying to use a different Guava. He's trying to
use the metastore client libraries for his version of Hive, but HiveContext,
when loading those libraries, says "everything under com.google.common should
be loaded from the parent class loader".
Except that when you build Spark with maven, the parent class loader does not
contain any of those classes, since they're shaded.
> Guava ClassLoading Issue When Using Different Hive Metastore Version
> --------------------------------------------------------------------
>
> Key: SPARK-11702
> URL: https://issues.apache.org/jira/browse/SPARK-11702
> Project: Spark
> Issue Type: Bug
> Affects Versions: 1.5.1
> Reporter: Joey Paskhay
>
> A Guava classloading error can occur when using a different version of the
> Hive metastore.
> Running the latest version of Spark at this time (1.5.1) and patched versions
> of Hadoop 2.2.0 and Hive 1.0.0. We set "spark.sql.hive.metastore.version" to
> "1.0.0" and "spark.sql.hive.metastore.jars" to
> "<path_to_hive>/lib/*:<output_of_hadoop_classpath_cmd>". When trying to
> launch the spark-shell, the sqlContext would fail to initialize with:
> {code}
> java.lang.ClassNotFoundException: java.lang.NoClassDefFoundError:
> com/google/common/base/Predicate when creating Hive client using classpath:
> <all the jars>
> Please make sure that jars for your version of hive and hadoop are included
> in the paths passed to SQLConfEntry(key = spark.sql.hive.metastore.jars,
> defaultValue=builtin, doc=...
> {code}
> We verified the Guava libraries are in the huge list of the included jars,
> but we saw that in the
> org.apache.spark.sql.hive.client.IsolatedClientLoader.isSharedClass method it
> seems to assume that *all* "com.google" (excluding "com.google.cloud")
> classes should be loaded from the base class loader. The Spark libraries seem
> to have *some* "com.google.common.base" classes shaded in but not all.
> See
> [https://mail-archives.apache.org/mod_mbox/spark-user/201511.mbox/%3CCAB51Vx4ipV34e=eishlg7bzldm0uefd_mpyqfe4dodbnbv9...@mail.gmail.com%3E]
> and its replies.
> The work-around is to add the guava JAR to the "spark.driver.extraClassPath"
> property.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
