[ https://issues.apache.org/jira/browse/MAPREDUCE-4961?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Allen Wittenauer updated MAPREDUCE-4961: ---------------------------------------- Labels: BB2015-05-TBR (was: ) > Map reduce running local should also go through ShuffleConsumerPlugin for > enabling different MergeManager implementations > ------------------------------------------------------------------------------------------------------------------------- > > Key: MAPREDUCE-4961 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4961 > Project: Hadoop Map/Reduce > Issue Type: Bug > Reporter: Jerry Chen > Assignee: Jerry Chen > Labels: BB2015-05-TBR > Attachments: MAPREDUCE-4961.patch, MAPREDUCE-4961.patch > > Original Estimate: 72h > Remaining Estimate: 72h > > MAPREDUCE-4049 provide the ability for pluggable Shuffle and MAPREDUCE-4080 > extends Shuffle to be able to provide different MergeManager implementations. > While using these pluggable features, I find that when a map reduce is > running locally, a RawKeyValueIterator was returned directly from a static > call of Merge.merge, which break the assumption that the Shuffle may provide > different merge methods although there is no copy phase for this situation. > The use case is when I am implementating a hash-based MergeManager, we don't > need sort in map side, while when running the map reduce locally, the > hash-based MergeManager will have no chance to be used as it goes directly to > Merger.merge. This makes the pluggable Shuffle and MergeManager incomplete. > So we need to move the code calling Merger.merge from Reduce Task to > ShuffleConsumerPlugin implementation, so that the Suffle implementation can > decide how to do the merge and return corresponding iterator. -- This message was sent by Atlassian JIRA (v6.3.4#6332)