[jira] [Commented] (MAPREDUCE-3060) Generic shuffle service
[ https://issues.apache.org/jira/browse/MAPREDUCE-3060?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13237899#comment-13237899 ] Avner BenHanoch commented on MAPREDUCE-3060: Please see MAPREDUCE-4049. > Generic shuffle service > --- > > Key: MAPREDUCE-3060 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3060 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: mrv2 >Affects Versions: 0.23.0 >Reporter: Luke Lu > Labels: shuffle > Fix For: 0.24.0 > > > When I was talking to Owen about MAPREDUCE-2600, we came across (again, > talked about it with Chris before) the shuffle dependency issue. NodeManager > currently has an implicit (hidden by the service plugin mechanism) dependency > of a specific version of mapreduce shuffle. While this works in many cases, > as long as we don't change shuffle headers and the usage of mapred security > tokens, it's a hack to make things work none the less. It's generally agreed > upon that nodemanager should only load generic services that are mapreduce > framework neutral. > In this particular case, the right solution seems to be a generic shuffle > handler that can serve data for a particular partition securely. The > ShuffleHandler currently only depends on mapreduce for task tokens and > shuffle header, which is only used for writing data, i.e., the shuffle > handler has no semantic dependency on mapreduce. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3060) Generic shuffle service
[ https://issues.apache.org/jira/browse/MAPREDUCE-3060?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13109872#comment-13109872 ] Milind Bhandarkar commented on MAPREDUCE-3060: -- +1 ! This makes a lot of optimized third party plugins possible. > Generic shuffle service > --- > > Key: MAPREDUCE-3060 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3060 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: mrv2 >Affects Versions: 0.23.0 >Reporter: Luke Lu > Labels: shuffle > Fix For: 0.24.0 > > > When I was talking to Owen about HADOOP-2600, we came across (again, talked > about it with Chris before) the shuffle dependency issue. NodeManager > currently has an implicit (hidden by the service plugin mechanism) dependency > of a specific version of mapreduce shuffle. While this works in many cases, > as long as we don't change shuffle headers and the usage of mapred security > tokens, it's a hack to make things work none the less. It's generally agreed > upon that nodemanager should only load generic services that are mapreduce > framework neutral. > In this particular case, the right solution seems to be a generic shuffle > handler that can serve data for a particular partition securely. The > ShuffleHandler currently only depends on mapreduce for task tokens and > shuffle header, which is only used for writing data, i.e., the shuffle > handler has no semantic dependency on mapreduce. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
