[
https://issues.apache.org/jira/browse/HADOOP-1965?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12531151
]
Runping Qi commented on HADOOP-1965:
------------------------------------
Having small enough map input by running more mappers will certainly avoid the
problem of spills.
On the other hand, you cannot make the input size too small, otherwise, the
overhead associated with
task startup and shuffling will become significant.
And in reality, it is very hard to choose the right number of mappers.
> Handle map output buffers better
> --------------------------------
>
> Key: HADOOP-1965
> URL: https://issues.apache.org/jira/browse/HADOOP-1965
> Project: Hadoop
> Issue Type: Improvement
> Components: mapred
> Reporter: Devaraj Das
> Assignee: Amar Kamat
>
> Today, the map task stops calling the map method while sort/spill is using
> the (single instance of) map output buffer. One improvement that can be done
> to improve performance of the map task is to have another buffer for writing
> the map outputs to, while sort/spill is using the first buffer.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
