[
https://issues.apache.org/jira/browse/HADOOP-1338?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12661665#action_12661665
]
Matei Zaharia commented on HADOOP-1338:
---------------------------------------
I think explicitly fetching multiple maps per connection is the best way to go.
We can control exactly what happens instead of relying on connection keep-alive
to work in the HTTP library and running into problems if this changes. In terms
of how many outputs to fetch, I believe we should take the same approach as
Dhruba's multiple-splits-per-map patch - don't say a fixed number of outputs,
but rather a fixed number of bytes (say up to 256 MB). If a job has small map
outputs, you should fetch more, and in the extreme, for something like
webdatascan, you might as well fetch all the outputs on a host in one go.
> Improve the shuffle phase by using the "connection: keep-alive" and doing
> batch transfers of files
> --------------------------------------------------------------------------------------------------
>
> Key: HADOOP-1338
> URL: https://issues.apache.org/jira/browse/HADOOP-1338
> Project: Hadoop Core
> Issue Type: Improvement
> Components: mapred
> Reporter: Devaraj Das
> Assignee: Jothi Padmanabhan
>
> We should do transfers of map outputs at the granularity of
> *total-bytes-transferred* rather than the current way of transferring a
> single file and then closing the connection to the server. A single
> TaskTracker might have a couple of map output files for a given reduce, and
> we should transfer multiple of them (upto a certain total size) in a single
> connection to the TaskTracker. Using HTTP-1.1's keep-alive connection would
> help since it would keep the connection open for more than one file transfer.
> We should limit the transfers to a certain size so that we don't hold up a
> jetty thread indefinitely (and cause timeouts for other clients).
> Overall, this should give us improved performance.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
