[
https://issues.apache.org/jira/browse/SPARK-9593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14653419#comment-14653419
]
Sean Owen commented on SPARK-9593:
----------------------------------
This isn't quite right: the method that the shim loader is looking for was
added in Hadoop 2.1, not 2.0. The CDH release is technically fine in this
regard. I'll post more on the PR.
> Hive ShimLoader loads wrong Hadoop shims when Spark is compiled against
> Hadoop 2.0.0-mr1-cdh4.1.1
> -------------------------------------------------------------------------------------------------
>
> Key: SPARK-9593
> URL: https://issues.apache.org/jira/browse/SPARK-9593
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 1.5.0
> Reporter: Cheng Lian
> Assignee: Cheng Lian
> Priority: Blocker
>
> Internally, Hive {{ShimLoader}} tries to load different versions of Hadoop
> shims by checking version information gathered from Hadoop jar files. If the
> major version number is 1, {{Hadoop20SShims}} will be loaded. Otherwise, if
> the major version number is 2, {{Hadoop23Shims}} will be chosen. However,
> CDH Hadoop versions like 2.0.0-mr1-cdh4.1.1 have 2 as major version number,
> but contain Hadoop 1 code. This confuses Hive {{ShimLoader}} and loads wrong
> version of shims.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
