[
https://issues.apache.org/jira/browse/MAPREDUCE-5221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13670163#comment-13670163
]
Tsuyoshi OZAWA commented on MAPREDUCE-5221:
-------------------------------------------
I've confirmed all tests are pass correctly. It's absolutely odd, because of
the difference of o.a.h.mapred.Reducer and o.a.h.mapreduce.Reduce as you
pointed out. Therefore, I've analysed the source code, and I've found some
problems:
1. There are no codes to test with MiniMRYarnCluster with launching reduce-side
combiner. TestMRAppWithCombiner seems to be enough, but reduce-side combiners
are actually not launched because data size is too small to launch combiners.
2. ShuffleConsumerPlugin.Contexts only accepts Class<? extends
o.a.h.mapred.Reducer> when constructing. It's needed to add new
ShuffleConsumerPlugin.Contexts which accepts Class<? extends
o.a.h.mapreduce.Reducer>. # Or, change huffleConsumerPlugin.Contexts to accepts
Class<?> and check the type with Java reflection API when constructing.
3. Shuffle#MergeManager only accepts Class<? extends o.a.h.mapred.Reducer>.
It's needed to add new MergeManager which accepts Class<? extends
o.a.h.mapreduce.Reducer>.
I'll create a subtasks to deal with these problems step by step.
> Reduce side Combiner is not used when using the new API
> -------------------------------------------------------
>
> Key: MAPREDUCE-5221
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5221
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Affects Versions: 2.0.4-alpha
> Reporter: Siddharth Seth
> Assignee: Tsuyoshi OZAWA
> Attachments: MAPREDUCE-5221.1.patch, MAPREDUCE-5221.2.patch,
> MAPREDUCE-5221.3.patch
>
>
> If a combiner is specified using o.a.h.mapreduce.Job.setCombinerClass - this
> will silently ignored on the reduce side since the reduce side usage is only
> aware of the old api combiner.
> This doesn't fail the job - since the new combiner key does not deprecate the
> old key.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
