[ https://issues.apache.org/jira/browse/MAPREDUCE-2454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13499171#comment-13499171 ]
Mariappan Asokan commented on MAPREDUCE-2454: --------------------------------------------- Hi Arun, Thanks for your comments. Regarding your comment on passing shuffle to merge: I did this with the following rationale: * Conceptually, merge can take its input from different types of sources; today we have two types: one is Shuffle and the other is from local files for a local job. Tomorrow, we may add a hybrid of Shuffle and local map output files(this will involve adding another method in the ReduceInputMerger) to avoid shuffling local map outputs for optimizing the performance. This new approach which decouples Shuffle and Merge is more flexible. * In current implementation, Shuffle which is supposed to transfer bytes from map outputs to the merger, is also returning a RawKeyValueIterator which in turn implies it is doing more than transferring bytes. -- Asokan > Allow external sorter plugin for MR > ----------------------------------- > > Key: MAPREDUCE-2454 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2454 > Project: Hadoop Map/Reduce > Issue Type: New Feature > Affects Versions: 2.0.0-alpha, 3.0.0, 2.0.2-alpha > Reporter: Mariappan Asokan > Assignee: Mariappan Asokan > Priority: Minor > Labels: features, performance, plugin, sort > Attachments: HadoopSortPlugin.pdf, HadoopSortPlugin.pdf, > KeyValueIterator.java, MapOutputSorterAbstract.java, MapOutputSorter.java, > mapreduce-2454-modified-code.patch, mapreduce-2454-modified-test.patch, > mapreduce-2454-new-test.patch, mapreduce-2454.patch, mapreduce-2454.patch, > mapreduce-2454.patch, mapreduce-2454.patch, mapreduce-2454.patch, > mapreduce-2454.patch, mapreduce-2454.patch, mapreduce-2454.patch, > mapreduce-2454.patch, mapreduce-2454.patch, mapreduce-2454.patch, > mapreduce-2454.patch, mapreduce-2454.patch, mapreduce-2454.patch, > mapreduce-2454.patch, mapreduce-2454.patch, mapreduce-2454.patch, > mapreduce-2454.patch, mapreduce-2454.patch, mapreduce-2454.patch, > mapreduce-2454.patch, mapreduce-2454-protection-change.patch, > mr-2454-on-mr-279-build82.patch.gz, MR-2454-trunkPatchPreview.gz, > ReduceInputSorter.java > > > Define interfaces and some abstract classes in the Hadoop framework to > facilitate external sorter plugins both on the Map and Reduce sides. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira