[
https://issues.apache.org/jira/browse/HADOOP-1965?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12536948
]
Amar Kamat commented on HADOOP-1965:
------------------------------------
Breakup of sort and combine+spill timings is attached in
1965_single_proc_150mb_gziped_breakup.png. The observation is as follows
* The time required for sort and spill is proportional to io.sort.mb.
* Sort timings increase rapidly as compared to combine+spill with an increase
in io.sort.mb.
Sort becomes a bottleneck as io.sort.mb is increased and performs badly than
spill causing the overall map timings to increase with io.sort.mb.
----
A test was made to see the effects of having multiple map tasks running on a
same node concurrently. The observation is that the performance gained by using
a threaded sort-spill over a sequential one is not very significant as compared
to running a single map task per node.
> Handle map output buffers better
> --------------------------------
>
> Key: HADOOP-1965
> URL: https://issues.apache.org/jira/browse/HADOOP-1965
> Project: Hadoop
> Issue Type: Improvement
> Components: mapred
> Reporter: Devaraj Das
> Assignee: Amar Kamat
> Attachments: 1965_single_proc_150mb_gziped.pdf
>
>
> Today, the map task stops calling the map method while sort/spill is using
> the (single instance of) map output buffer. One improvement that can be done
> to improve performance of the map task is to have another buffer for writing
> the map outputs to, while sort/spill is using the first buffer.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
