[
https://issues.apache.org/jira/browse/PIG-3781?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Cheolsoo Park updated PIG-3781:
-------------------------------
Attachment: PIG-3781-1.patch
In the attached patch, I changed the condition in Tez CombinerOptimizer for
detecting co-group as follows-
From:
{code}
if (predecessors.size()>1) {
{code}
To:
{code}
if (to.isCogroup()) {
{code}
One corner case that the old condition fails is when group-by + algebraic udf
are followed by join (e.g. TestTezCompiler.testReplicatedJoinInReducer). In
this case, we want to add combiner to the output edge of group-by. In MR, this
cannot happen because join never has more than one inputs, so checking the
number of predecessors was fine.
Now I am explicitly checking whether the reduce vertex is co-group or not. I
also added a new test case to TestTezCompiler to verify that the combiner is
disabled for co-group.
> Fix TestTezCompiler#testReplicatedJoinInReducer
> -----------------------------------------------
>
> Key: PIG-3781
> URL: https://issues.apache.org/jira/browse/PIG-3781
> Project: Pig
> Issue Type: Sub-task
> Components: tez
> Affects Versions: tez-branch
> Reporter: Cheolsoo Park
> Assignee: Cheolsoo Park
> Fix For: tez-branch
>
> Attachments: PIG-3781-1.patch
>
>
> TestTezCompiler#testReplicatedJoinInReducer is broken because of PIG-3759. We
> need to fix it.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
