[
https://issues.apache.org/jira/browse/HADOOP-8462?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jason Lowe updated HADOOP-8462:
-------------------------------
Status: Open (was: Patch Available)
Thanks for the updates, Govind. One thing I missed before is that there is a
mixture of spaces and tabs in the patch. The tabs should be removed and
replaced with spaces so it lines up correctly regardless of the tab stop
setting with which it's being viewed. Could you also post the corresponding
patch for branch-0.23?
I gave the patch a test drive on a pseudo-distributed cluster, testing building
with and without the bzip2-devel package installed, running the unit tests with
the native bzip2 available, and verifying require.bzip2 works as intended. I
also ran some randomtextwriter jobs with a relatively small amount of data
(10MB). Using the native bzip2 codec for the output format codec was around
6.5x the job runtime performance of the pure Java codec for that (admittedly
not real-world) scenario.
> Native-code implementation of bzip2 codec
> -----------------------------------------
>
> Key: HADOOP-8462
> URL: https://issues.apache.org/jira/browse/HADOOP-8462
> Project: Hadoop Common
> Issue Type: Improvement
> Components: io
> Affects Versions: 0.23.1
> Reporter: Govind Kamat
> Assignee: Govind Kamat
> Attachments: HADOOP-8462-2.0.2a.1.patch, HADOOP-8462-2.0.2a.patch,
> HADOOP-8462-trunk.1.patch, HADOOP-8462-trunk.1.patch,
> HADOOP-8462-trunk.patch, HADOOP-8462-trunk.patch, HADOOP-8462-trunk.patch
>
> Original Estimate: 672h
> Remaining Estimate: 672h
>
> The bzip2 codec supplied with Hadoop is currently available only as a Java
> implementation. A version that uses the system bzip2 library can provide
> improved performance and a better memory footprint. This will also make it
> feasible to utilize alternative bzip2 libraries that may perform better for
> specific jobs.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
