[
https://issues.apache.org/jira/browse/HADOOP-11064?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14126835#comment-14126835
]
Steve Loughran commented on HADOOP-11064:
-----------------------------------------
I'd like to see this patch not reverting any of the changes, but offering the
existing API calls. On windows machines, the native lib is near-essential for
functionality; on linux it is the only way to get performance on IO. With
things like unix-socket shortcuts and native encryption, this only gets worse.
It shows up on YARN apps more than anywhere else because these apps are not
things built by bigtop or anyone else and pre-installed on the cluster. They
are apps submitted remotely —and to avoid problems with transitive dependencies
&c, they are moving towards a "submit all the JARs" strategy. Twill does this,
Tez does this, SLIDER-330 shows where it is on my todo list. This stops my code
being brittle against versions, but it only works if the right JAR is on the CP
We can't (currently) mandate that YARN apps know which platform lib to use, to
include it in their binaries and upload to it, so we do need a good story here.
Long term, I think
# some versioning like the current patch can work
# I think for windows, we should at least have a winutils.jar JAR that includes
the windows one. As chris points out, packaging gets complex here. Unless,
perhaps, it gets teased out and made its own self-contained hadop-tools project
that gets pre-published, potentially on a different release schedule.
Anyway, right now I'd be happier with a recovery of the old methods, while we
think of a good policy and how to implement it.
> UnsatisifedLinkError with hadoop 2.4 JARs on hadoop-2.6 due NativeCRC32
> method changes
> --------------------------------------------------------------------------------------
>
> Key: HADOOP-11064
> URL: https://issues.apache.org/jira/browse/HADOOP-11064
> Project: Hadoop Common
> Issue Type: Bug
> Components: native
> Affects Versions: 2.6.0
> Environment: Hadoop 2.6 cluster, trying to run code containing hadoop
> 2.4 JARs
> Reporter: Steve Loughran
> Assignee: Colin Patrick McCabe
> Priority: Blocker
> Attachments: HADOOP-11064.001.patch
>
>
> The private native method names and signatures in {{NativeCrc32}} were
> changed in HDFS-6561 ... as a result hadoop-common-2.4 JARs get unsatisifed
> link errors when they try to perform checksums.
> This essentially stops Hadoop 2.4 applications running on Hadoop 2.6 unless
> rebuilt and repackaged with the hadoop- 2.6 JARs
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
