[
https://issues.apache.org/jira/browse/MAPREDUCE-1891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12881719#action_12881719
]
Xiao Kang commented on MAPREDUCE-1891:
--------------------------------------
Compression of intermidate result of map output is used to decrease the local
disk io and network bandwith usage. However, in shuffle phase, shuffled data is
decompressed immediately and more memeoy is used. If the shuffled data is
keeped compressed until merging to disk, less memory is used for the same data
set. Then more shuffled data can fit in the same shuffle buffer and less disk
merge is needed.
Any suggestions?
> Disk io of shuffle phase may decrease if shuffled data not decompressed
> -----------------------------------------------------------------------
>
> Key: MAPREDUCE-1891
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1891
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: tasktracker
> Reporter: Xiao Kang
>
> Current shuffled data is decompressed immediately. It may use much less
> memory for shuffle buffer if the data is not decompressed until merging to
> disk.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
