[
https://issues.apache.org/jira/browse/HADOOP-1338?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12658036#action_12658036
]
Jothi Padmanabhan commented on HADOOP-1338:
-------------------------------------------
I did another test to see if pulling in more data per connection is beneficial.
I just ran a simple client/server application on sockets. In the following
table, for each transfer, a connection was created and destroyed.
Here are the results
A. When nodes are in the same rack
||Transfer Size (MB)||Num Transfers||Total Time (seconds)||
|50|200|123
|100|100|109
|200|50|101
|400|25|97
B. When the nodes were on a different rack
||Transfer Size (MB)||Num Transfers||Total Time (seconds)||
|50|200|158
|100|100|142
|200|50|134
|400|25|130
>From these results it does appear that there could be a small but definite
>advantage in bunching the outputs, especially when each output is small.
Thoughts?
> Improve the shuffle phase by using the "connection: keep-alive" and doing
> batch transfers of files
> --------------------------------------------------------------------------------------------------
>
> Key: HADOOP-1338
> URL: https://issues.apache.org/jira/browse/HADOOP-1338
> Project: Hadoop Core
> Issue Type: Improvement
> Components: mapred
> Reporter: Devaraj Das
>
> We should do transfers of map outputs at the granularity of
> *total-bytes-transferred* rather than the current way of transferring a
> single file and then closing the connection to the server. A single
> TaskTracker might have a couple of map output files for a given reduce, and
> we should transfer multiple of them (upto a certain total size) in a single
> connection to the TaskTracker. Using HTTP-1.1's keep-alive connection would
> help since it would keep the connection open for more than one file transfer.
> We should limit the transfers to a certain size so that we don't hold up a
> jetty thread indefinitely (and cause timeouts for other clients).
> Overall, this should give us improved performance.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
