[
https://issues.apache.org/jira/browse/MAPREDUCE-4808?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13526795#comment-13526795
]
Alejandro Abdelnur commented on MAPREDUCE-4808:
-----------------------------------------------
Arun, what do you exactly mean?
Both Merger and MergeManager are classes.
The reducer side of logic in the Merger class is used only in 2 places, within
the MergeManager (used by the Shuffle) and in the ReduceTask (for the local
case).
This patch is moving the later use into the MergeManager as well. By doing
this all reduce merge logic, for the local and the distributed case, is
encapsulated in the MergerManager.
Then, when an alternate implementation is provided, it can handle both cases,
local and distributed case.
The resulting interface being introduced as ReduceInputMerger is quite simple:
{code}
public void init(Context<K, V> reduceMergerContext);
public void waitForResource() throws InterruptedException;
public MapOutput<K, V> reserve(TaskAttemptID mapId, long requestedSize,
int fetcher) throws IOException;
public RawKeyValueIterator close() throws Throwable;
// To merge files created for a local job.
public RawKeyValueIterator mergeLocalFiles(Path localFiles[])
throws IOException;
{code}
I think this is much simpler than trying modify things in the Merger, given
that the merger is not directly used by the Shuffle, but through the
MergeManager.
> Allow reduce-side merge to be pluggable
> ---------------------------------------
>
> Key: MAPREDUCE-4808
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4808
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Affects Versions: 2.0.2-alpha
> Reporter: Arun C Murthy
> Assignee: Mariappan Asokan
> Fix For: 2.0.3-alpha
>
> Attachments: COMBO-mapreduce-4809-4812-4808.patch,
> mapreduce-4808.patch
>
>
> Allow reduce-side merge to be pluggable for MAPREDUCE-2454
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
