[
http://issues.apache.org/jira/browse/HADOOP-538?page=comments#action_12438481 ]
Doug Cutting commented on HADOOP-538:
-------------------------------------
> we need a class that is responsible for trying to load libhadoop.so and
> having fall back replacements
+1
I question whether DF belongs here. Each thing here requires two
implementations. So we should only do it in places where it is critical.
We've found the improvements of native codecs to be dramatic. Do we think that
DF is causing performance problems? Because we will still need the non-native
version of DF as a fallback, for OSes and architectures that we won't provide
pre-built binaries for in releases.
It would also be good to minimize the number of libhadoop binaries we commit to
subversion. I'd argue that we only check in libhadoop-linux-i386.so. Even
that will be impossible to rebuild in nightly builds, since those are made on
solaris x86. We will require that any time anything in the native code is
changed, all pre-built binaries be re-generated as a part of the commit of the
source changes. Since linux-i386 is both the dominant development and
deployment environment, I think it makes sense to ship with it alone pre-built.
> Implement a nio's 'direct buffer' based wrapper over zlib to improve
> performance of java.util.zip.{De|In}flater as a 'custom codec'
> -----------------------------------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-538
> URL: http://issues.apache.org/jira/browse/HADOOP-538
> Project: Hadoop
> Issue Type: Improvement
> Affects Versions: 0.6.1
> Reporter: Arun C Murthy
> Assigned To: Arun C Murthy
> Fix For: 0.7.0
>
> Attachments: HADOOP-538.patch, HADOOP-538_benchmarks.tgz
>
>
> There has been more than one instance where java.util.zip's {De|In}flater
> classes perform unreliably, a simple wrapper over zlib-1.2.3 (latest stable)
> using java.nio.ByteBuffer (i.e. direct buffers) should go a long way in
> alleviating these woes.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
