[
https://issues.apache.org/jira/browse/MAPREDUCE-2786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13436967#comment-13436967
]
Konstantin Shvachko commented on MAPREDUCE-2786:
------------------------------------------------
That is a good thing to have an opportunity to benchmark with compression.
Couple suggestions.
# Move all compression configuration logic, including reflections and the cc
variable all the way to {{IOMapperBase.configure()}}. Otherwise all this small
actions will be counted as execution time.
# You should not work separately with compressed and non-compressed streams
inside doIO(). Same {{out}} or {{in}} variables should just point to compressed
or not compressed streams. Nesting streams is a regular practice.
# {{getCompression()}} is not used anywhere, should be removed.
# You use {{test.compression}} to get the codec class and
{{test.io.compression.class}} to set it. How is going to work? You should make
two constants with the property and the default value and use them.
# AppendMapper is not covered. It should be the same as others. Moving the
conig logic into {{IOMapperBase}} should make it easy.
> TestDFSIO should also test compression reading/writing from command-line.
> -------------------------------------------------------------------------
>
> Key: MAPREDUCE-2786
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2786
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: benchmarks
> Affects Versions: 2.0.0-alpha
> Reporter: Plamen Jeliazkov
> Priority: Minor
> Labels: newbie
> Fix For: 2.1.0-alpha
>
> Attachments: MAPREDUCE-2786.patch
>
> Original Estimate: 36h
> Remaining Estimate: 36h
>
> I thought it might be beneficial to simply alter the code of TestDFSIO to
> accept any compression codec class and allow testing for compression by a
> command line argument instead of having to change the config file everytime.
> Something like "-compression" would do.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
