[
https://issues.apache.org/jira/browse/PIG-4002?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14039695#comment-14039695
]
Daniel Dai commented on PIG-4002:
---------------------------------
Checked with [~thejas], the primary reason for combiner is to use in the reduce
side:
* In map side, POPartialAgg will be used for aggregation
* In reduce side, combiner will be used for aggregation if we have large number
of maps
Ideally we want a flag to only use combiner in reduce side but not map side.
However, [~rohini] point to me combiner in reduce side is not working
(MAPREDUCE-5221). As long as the bug is still there, combiner will not be
useful when map side aggregation is used, and we shall disable it.
> Disable combiner when map-side aggregation is used
> --------------------------------------------------
>
> Key: PIG-4002
> URL: https://issues.apache.org/jira/browse/PIG-4002
> Project: Pig
> Issue Type: Improvement
> Affects Versions: 0.13.0
> Reporter: Travis Woodruff
> Assignee: Travis Woodruff
> Priority: Minor
> Attachments: PIG-4002-1.patch
>
>
> This may be controversial, so I'd like others' opinions on this.
> It is not currently possible to disable the combiner and use map-side
> aggregation at the same time. This is a problematic since map-side
> aggregation effectively combines in the mapper, so running the combiner adds
> expensive combiner execution (combiner requires deserialization &
> reserialization) for little to no value.
> PIG-2829 had a patch to disable the combiner when map-side aggregation is
> used (along with some other changes). This was never integrated because the
> map-side aggregation code was redone while this was in progress.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
