[
https://issues.apache.org/jira/browse/SPARK-2420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14056751#comment-14056751
]
Xuefu Zhang commented on SPARK-2420:
------------------------------------
[~rxin] upgrading guava to 14 is no problem to Hive. However, major hadoop
components are still on 11. Upgrading all to 14 doesn't seem feasible.
[~srowen] Based on our experience, servlet 3.0 is a problem to Hive. This
probably due to the fact that Hive and its sub-component (such as hcatalog) use
servlet 2.5 for the webui and conflicts arise when Spark library is loaded to
Hive. In general, it's a problem if multiple versions of the same library are
loaded to JVM, especially by the same class loader. The fact that shading those
jars in Spark solved the problem further proves the problem. Let me know if you
like know a stack trace of the problem when the jars are not shaded.
To make integration easier, it would be nice if Spark can keep the versions of
these popular modules consistent with the rest of Hadoop components.
> Change Spark build to minimize library conflicts
> ------------------------------------------------
>
> Key: SPARK-2420
> URL: https://issues.apache.org/jira/browse/SPARK-2420
> Project: Spark
> Issue Type: Wish
> Components: Build
> Affects Versions: 1.0.0
> Reporter: Xuefu Zhang
> Attachments: spark_1.0.0.patch
>
>
> During the prototyping of HIVE-7292, many library conflicts showed up because
> Spark build contains versions of libraries that's vastly different from
> current major Hadoop version. It would be nice if we can choose versions
> that's in line with Hadoop or shading them in the assembly. Here are the wish
> list:
> 1. Upgrade protobuf version to 2.5.0 from current 2.4.1
> 2. Shading Spark's jetty and servlet dependency in the assembly.
> 3. guava version difference. Spark is using a higher version. I'm not sure
> what's the best solution for this.
> The list may grow as HIVE-7292 proceeds.
> For information only, the attached is a patch that we applied on Spark in
> order to make Spark work with Hive. It gives an idea of the scope of changes.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
