Rack-aware Shuffle
------------------
Key: HADOOP-1721
URL: https://issues.apache.org/jira/browse/HADOOP-1721
Project: Hadoop
Issue Type: Improvement
Components: mapred
Reporter: Arun C Murthy
Assignee: Arun C Murthy
Fix For: 0.15.0
We could try and experiment with *rack-aware* scheduling of fetches
per-reducer. Given the disparities between in-rack and off-rack bandwidth it
could be a improvement to do something along these lines:
{noformat}
if (no. of known map-output locations > than no. of copier threads) {
try to schedule 75% of copies off-rack
try schedule 25% of copies in-rack
}
{noformat}
This could lead to better utilization of both in-rack & switch b/w...
Clearly we want to schedule more cross-switch than in-rack since off-rack
copies will take significantly more time; hence the 75-25 split.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
