[
https://issues.apache.org/jira/browse/HADOOP-4640?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Johan Oskarsson updated HADOOP-4640:
------------------------------------
Attachment: HADOOP-4640.patch
Replaced the TreeSet with long[], fixed the incorrect checksum count, fixed the
indexer loop termination.
As for the close() I did as suggested, although it rubs me the wrong way to
read all those bytes without needing to. I guess the practical performance
impact will be minimal though.
> Add ability to split text files compressed with lzo
> ---------------------------------------------------
>
> Key: HADOOP-4640
> URL: https://issues.apache.org/jira/browse/HADOOP-4640
> Project: Hadoop Core
> Issue Type: Improvement
> Components: io, mapred
> Reporter: Johan Oskarsson
> Assignee: Johan Oskarsson
> Priority: Trivial
> Fix For: 0.20.0
>
> Attachments: HADOOP-4640.patch, HADOOP-4640.patch, HADOOP-4640.patch
>
>
> Right now any file compressed with lzop will be processed by one mapper. This
> is a shame since the lzo algorithm would be very suitable for large log files
> and similar common hadoop data sets. The compression rate is not the best out
> there but the decompression speed is amazing. Since lzo writes compressed
> data in blocks it would be possible to make an input format that can split
> the files.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
