[
https://issues.apache.org/jira/browse/HADOOP-2402?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Douglas updated HADOOP-2402:
----------------------------------
Attachment: 2402-1.patch
This patch reworks the lzo codec to fill its nio buffer before spilling to the
native code. It effects the following results for 5GB of random text:
Write:
|| Format || Type || Time (s from trunk, pcnt change) || Compression (% from
uncompressed) ||
| SEQ | BLK | -51 (-31.1%) | 54.9% (23.7% using trunk) |
| SEQ | REC | +5 (+2.7%) | -3.6% (-3.6% using trunk) |
| TXT | | -96 (-63.9%) | 54.8% (-12.3% using trunk) |
Read:
|| Format || Type || Time (s from trunk, pcnt change) ||
| SEQ | BLK | +7 (+8.6%) |
| SEQ | REC | +8 (+5.6%) |
| TXT | | -83 (-39.1%) |
The default extension should probably be something other than .lzo, since we're
not compatible with lzop.
> Lzo compression compresses each write from TextOutputFormat
> -----------------------------------------------------------
>
> Key: HADOOP-2402
> URL: https://issues.apache.org/jira/browse/HADOOP-2402
> Project: Hadoop
> Issue Type: Bug
> Components: io, mapred, native
> Reporter: Chris Douglas
> Fix For: 0.16.0
>
> Attachments: 2402-0.patch, 2402-1.patch
>
>
> Outputting with TextOutputFormat and Lzo compression generates a file such
> that each key, tab delimiter, and value are compressed separately.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
