Generic shuffle service
-----------------------
Key: MAPREDUCE-3060
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3060
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: mrv2
Affects Versions: 0.23.0
Reporter: Luke Lu
Fix For: 0.24.0
When I was talking to Owen about HADOOP-2600, we came across (again, talked
about it with Chris before) the shuffle dependency issue. NodeManager currently
has an implicit (hidden by the service plugin mechanism) dependency of a
specific version of mapreduce shuffle. While this works in many cases, as long
as we don't change shuffle headers and the usage of mapred security tokens,
it's a hack to make things work none the less. It's generally agreed upon that
nodemanager should only load generic services that are mapreduce framework
neutral.
In this particular case, the right solution seems to be a generic shuffle
handler that can serve data for a particular partition securely. The
ShuffleHandler currently only depends on mapreduce for task tokens and shuffle
header, which is only used for writing data, i.e., the shuffle handler has no
semantic dependency on mapreduce.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
