[
https://issues.apache.org/jira/browse/MAPREDUCE-4812?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13526802#comment-13526802
]
Mariappan Asokan commented on MAPREDUCE-4812:
---------------------------------------------
Hi Arun,
Sorry I did not get back sooner. The intention of {{ReduceInputMerger}}
interface is to have a pluggable {{MergeManager}} implementation. For a
non-local job, {{Shuffle}} and {{MergeManager}} interact and synchronize with
each other using the three methods {{waitForInMemoryMerge(),}} {{reserve(),}}
and {{close()}}. So in order to use the {{Shuffle}} these methods are captured
in {{ReduceInputMerger}} interface. I renamed {{waitForInMemoryMerge()}} to a
generic name {{waitForResource()}} since the plugin implementation may not have
the concept of in-memory merge.
Since the return value from {{reserve()}} is {{MapOutput}}, I did some
refactoring of {{MapOutput}} so that plugin can return its own implementation
of it. I kept the refactoring done on {{MapOutput}} in MAPREDUCE-4808. With
just MAPREDUCE-4812, an external plugin is not possible, but it has the core
part of the concepts so that it is easy to review just {{ReduceInputMerger}}
design. Similarly, for a local job the input is coming from local files. I
enhanced {{ReduceInputMerger}} with one more method for this. It is also kept
in MAPREDUCE-4808.
Hope I explained well. Please let me know if you have any more questions.
Thanks.
-- Asokan
> Create reduce input merger plugin in ReduceTask.java and pass it to Shuffle
> ---------------------------------------------------------------------------
>
> Key: MAPREDUCE-4812
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4812
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Affects Versions: 2.0.2-alpha
> Reporter: Mariappan Asokan
> Assignee: Mariappan Asokan
> Fix For: 2.0.3-alpha
>
> Attachments: COMBO-mapreduce-4809-4812.patch,
> COMBO-mapreduce-4809-4812.patch, mapreduce-4812.patch, mapreduce-4812.patch,
> mapreduce-4812.patch, mapreduce-4812.patch, mapreduce-4812.patch
>
>
> This is part of MAPREDUCE-2454. This further breaks down MAPREDUCE-4808
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira