[
https://issues.apache.org/jira/browse/HADOOP-1338?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12658150#action_12658150
]
Devaraj Das commented on HADOOP-1338:
-------------------------------------
I would like to add one point to this (which is already what we do today) -
i.e., stall the copier (drop the connection) when we find that a map output
fits the bill for an inMemory shuffle but we don't presently have enough space
in the inmemory shuffle area. In our earlier benchmarks we found that we gain a
lot by stalling the copier if such a situation happens (mainly due to the costs
of the disk IO).
Also, in such a case, it might be good for the tasktracker to send available
outputs in the ascending order of their sizes. That way, the reducer gets a
better chance of fitting more outputs in memory before dropping the connection,
if at all.
Owen, are you suggesting that we don't do the copier stalling at all?
Thoughts?
> Improve the shuffle phase by using the "connection: keep-alive" and doing
> batch transfers of files
> --------------------------------------------------------------------------------------------------
>
> Key: HADOOP-1338
> URL: https://issues.apache.org/jira/browse/HADOOP-1338
> Project: Hadoop Core
> Issue Type: Improvement
> Components: mapred
> Reporter: Devaraj Das
>
> We should do transfers of map outputs at the granularity of
> *total-bytes-transferred* rather than the current way of transferring a
> single file and then closing the connection to the server. A single
> TaskTracker might have a couple of map output files for a given reduce, and
> we should transfer multiple of them (upto a certain total size) in a single
> connection to the TaskTracker. Using HTTP-1.1's keep-alive connection would
> help since it would keep the connection open for more than one file transfer.
> We should limit the transfers to a certain size so that we don't hold up a
> jetty thread indefinitely (and cause timeouts for other clients).
> Overall, this should give us improved performance.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
