[
https://issues.apache.org/jira/browse/HADOOP-11064?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14128027#comment-14128027
]
Chris Nauroth commented on HADOOP-11064:
----------------------------------------
bq. I agree, it would be great to see the Windows build of hadoop.dll use CMake.
Earlier today, I filed HADOOP-11080 to track this.
bq. It would be better to solve it inside YARN, than to force a draconian
internal API compatibility policy on to libhadoop.
I can't think of a feature request that can be made of YARN for this.
hadoop-common.jar essentially does not publish a versioned release of its
corresponding native component. Asking YARN to figure out the right matching
native build at submission time would be brittle. Asking the applications to
bundle the matching native build version themselves pushes the build complexity
that we haven't solved down to our downstream consumers.
Hopefully I didn't imply that YARN could figure out how to distribute the
dependency. What I really wanted to convey is that hadoop-common-<version>.jar
is tightly coupled to a specific version of libhadoop.so, and if libhadoop.so
was inside the jar, then it would solve the problem for all applications that
want to bundle a specific Hadoop jar version and submit that as a YARN
application.
bq. Also, what are the "old" function signatures? The ones for Hadoop 2.4? 2.5?
2.5.1?
That's a fair point. The specific bug report here involves a 2.4.0 YARN
application submitted to a 2.6.0 YARN cluster, but I don't know what other
incompatibilities we'd expose with other version combinations.
bq. I am + 1 for v3 of the patch... and based on previous conversations, I
think most people would be OK with adding versioning to libhadoop.
I heard a general desire for .so versioning. (The .so version has been at
1.0.0 forever as far as I can tell.) I could be wrong, but I don't think I
heard general consensus for the scheme implemented in the patch. This is not
standard .so versioning. Instead, it's a whole new library name with every
release, even minor releases. Just so everyone is on the same page, here is
what we get from a release with this patch:
{code}
> ll hadoop-3.0.0-SNAPSHOT/lib/native/libhadoop*.so*
lrwxrwxrwx 1 cnauroth 33 Sep 9 20:49
hadoop-3.0.0-SNAPSHOT/lib/native/libhadoop-3.0.0-SNAPSHOT.so ->
libhadoop-3.0.0-SNAPSHOT.so.1.0.0*
-rwxrwxr-x 1 cnauroth 763K Sep 9 20:49
hadoop-3.0.0-SNAPSHOT/lib/native/libhadoop-3.0.0-SNAPSHOT.so.1.0.0*
{code}
This means that during upgrades, the cluster administrator or the distro
packaging must maintain all prior versions. Otherwise, upgrading a cluster
from 2.4.0 to 2.4.1 risks link errors for applications that happen to bundle
the 2.4.0 jar. In this sense, I'm concerned that the patch actually makes the
situation worse. It certainly changes the maintenance and deployment model.
With consideration of that, are you still +1?
bq. I apologize again for being difficult here...
No apologies necessary. This is all healthy discussion. You've presented fair
counter-points and handled the discussion respectfully. I hope I've handled my
end of the discussion as well as you have. :-)
> UnsatisifedLinkError with hadoop 2.4 JARs on hadoop-2.6 due NativeCRC32
> method changes
> --------------------------------------------------------------------------------------
>
> Key: HADOOP-11064
> URL: https://issues.apache.org/jira/browse/HADOOP-11064
> Project: Hadoop Common
> Issue Type: Bug
> Components: native
> Affects Versions: 2.6.0
> Environment: Hadoop 2.6 cluster, trying to run code containing hadoop
> 2.4 JARs
> Reporter: Steve Loughran
> Assignee: Colin Patrick McCabe
> Priority: Blocker
> Attachments: HADOOP-11064.001.patch, HADOOP-11064.002.patch,
> HADOOP-11064.003.patch
>
>
> The private native method names and signatures in {{NativeCrc32}} were
> changed in HDFS-6561 ... as a result hadoop-common-2.4 JARs get unsatisifed
> link errors when they try to perform checksums.
> This essentially stops Hadoop 2.4 applications running on Hadoop 2.6 unless
> rebuilt and repackaged with the hadoop- 2.6 JARs
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
