[ https://issues.apache.org/jira/browse/MAPREDUCE-259?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13666927#comment-13666927 ]
nemon lou commented on MAPREDUCE-259: ------------------------------------- MAPREDUCE-2038 is doing the same thing,but is not updated for a long time,too. > Rack-aware Shuffle > ------------------ > > Key: MAPREDUCE-259 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-259 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Reporter: Arun C Murthy > Assignee: Arun C Murthy > > We could try and experiment with *rack-aware* scheduling of fetches > per-reducer. Given the disparities between in-rack and off-rack bandwidth it > could be a improvement to do something along these lines: > {noformat} > if (no. of known map-output locations > than no. of copier threads) { > try to schedule 75% of copies off-rack > try schedule 25% of copies in-rack > } > {noformat} > This could lead to better utilization of both in-rack & switch b/w... > Clearly we want to schedule more cross-switch than in-rack since off-rack > copies will take significantly more time; hence the 75-25 split. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira