[
https://issues.apache.org/jira/browse/MAPREDUCE-2910?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13096818#comment-13096818
]
Binglin Chang commented on MAPREDUCE-2910:
------------------------------------------
LzoCodec: 2byte EOF marker + 4 byte checksum -> 14 byte compressed data + 4
byte checksum
GzipCodec: 2byte EOF marker + 4 byte checksum -> 26 byte compressed data + 4
byte checksum
Empty segments don't have any bytes, thus the seek & read in MapOutputServlet
can also be saved.
This optimization is only for extreme cases, I often see large proportion(90%)
of empty segments in vary large jobs(particularly with map side filter) in our
cluster, this is partially because of bad configuration or bad partitioner, but
tuning a partitioner or key distribution sometimes is non trivial for user.
> Allow empty MapOutputFile segments
> ----------------------------------
>
> Key: MAPREDUCE-2910
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2910
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: task, tasktracker
> Affects Versions: 0.20.2, 0.23.0
> Reporter: Binglin Chang
> Priority: Minor
> Fix For: 0.23.0
>
>
> As the scale of cluster and job get larger, we see a lot of empty partitions
> in MapOutputFile due to large reduce numbers or partition skew. When map
> output compression is enabled, empty map output partitions gets larger & has
> additional compressor/decompressor initialization overhead.
> This can be optimized by allowing empty MapOutputFile segments, where the
> rawLength & partLength of IndexRecord all equal to 0. Corresponding support
> need to be added to IFile reader, writer, and reduce shuffle copier.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
