[jira] Updated: (PIG-834) incorrect plan when algebraic functions are nested
[
https://issues.apache.org/jira/browse/PIG-834?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Thejas M Nair updated PIG-834:
--
Description:
a = load 'students.txt' as (c1,c2,c3,c4);
c = group a by c2;
f = foreach c generate COUNT(org.apache.pig.builtin.Distinct($1.$2));
Notice that Distinct udf is missing in Combiner and reduce stage. As a result
distinct does not function, and incorrect results are produced.
Distinct should have been evaluated in the 3 stages and output of Distinct
should be given to COUNT in reduce stage.
{code}
# Map Reduce Plan
#--
MapReduce node 1-122
Map Plan
Local Rearrange[tuple]{bytearray}(false) - 1-139
| |
| Project[bytearray][1] - 1-140
|
|---New For Each(false,false)[bag] - 1-127
| |
| POUserFunc(org.apache.pig.builtin.COUNT$Initial)[tuple] - 1-125
| |
| |---POUserFunc(org.apache.pig.builtin.Distinct)[bag] - 1-126
| |
| |---Project[bag][2] - 1-123
| |
| |---Project[bag][1] - 1-124
| |
| Project[bytearray][0] - 1-133
|
|---Pre Combiner Local Rearrange[tuple]{Unknown} - 1-141
|
|---Load(hdfs://wilbur11.labs.corp.sp1.yahoo.com/user/tejas/students.txt:org.apache.pig.builtin.PigStorage)
- 1-111
Combine Plan
Local Rearrange[tuple]{bytearray}(false) - 1-143
| |
| Project[bytearray][1] - 1-144
|
|---New For Each(false,false)[bag] - 1-132
| |
| POUserFunc(org.apache.pig.builtin.COUNT$Intermediate)[tuple] - 1-130
| |
| |---Project[bag][0] - 1-135
| |
| Project[bytearray][1] - 1-134
|
|---POCombinerPackage[tuple]{bytearray} - 1-137
Reduce Plan
Store(fakefile:org.apache.pig.builtin.PigStorage) - 1-121
|
|---New For Each(false)[bag] - 1-120
| |
| POUserFunc(org.apache.pig.builtin.COUNT$Final)[long] - 1-119
| |
| |---Project[bag][0] - 1-136
|
|---POCombinerPackage[tuple]{bytearray} - 1-145
Global sort: false
{code}
was:
a = load 'students.txt' as (c1,c2,c3,c4);
c = group a by c2;
f = foreach c generate COUNT(org.apache.pig.builtin.Distinct($1.$2));
Notice that Distinct udf is missing in Combiner and reduce stage. As a result
distinct does not function, and incorrect results are produced.
Distinct should have been evaluated in the 3 stages and output of Distinct
should be given to COUNT in reduce stage.
# Map Reduce Plan
#--
MapReduce node 1-122
Map Plan
Local Rearrange[tuple]{bytearray}(false) - 1-139
| |
| Project[bytearray][1] - 1-140
|
|---New For Each(false,false)[bag] - 1-127
| |
| POUserFunc(org.apache.pig.builtin.COUNT$Initial)[tuple] - 1-125
| |
| |---POUserFunc(org.apache.pig.builtin.Distinct)[bag] - 1-126
| |
| |---Project[bag][2] - 1-123
| |
| |---Project[bag][1] - 1-124
| |
| Project[bytearray][0] - 1-133
|
|---Pre Combiner Local Rearrange[tuple]{Unknown} - 1-141
|
|---Load(hdfs://wilbur11.labs.corp.sp1.yahoo.com/user/tejas/students.txt:org.apache.pig.builtin.PigStorage)
- 1-111
Combine Plan
Local Rearrange[tuple]{bytearray}(false) - 1-143
| |
| Project[bytearray][1] - 1-144
|
|---New For Each(false,false)[bag] - 1-132
| |
| POUserFunc(org.apache.pig.builtin.COUNT$Intermediate)[tuple] - 1-130
| |
| |---Project[bag][0] - 1-135
| |
| Project[bytearray][1] - 1-134
|
|---POCombinerPackage[tuple]{bytearray} - 1-137
Reduce Plan
Store(fakefile:org.apache.pig.builtin.PigStorage) - 1-121
|
|---New For Each(false)[bag] - 1-120
| |
| POUserFunc(org.apache.pig.builtin.COUNT$Final)[long] - 1-119
| |
| |---Project[bag][0] - 1-136
|
|---POCombinerPackage[tuple]{bytearray} - 1-145
Global sort: false
> incorrect plan when algebraic functions are nested
> --
>
> Key: PIG-834
> URL: https://issues.apache.org/jira/browse/PIG-834
> Project: Pig
> Issue Type: Bug
> Components: impl
>Reporter: Thejas M Nair
>Priority: Critical
>
> a = load 'students.txt' as (c1,c2,c3,c4);
> c = group a by c2;
> f = foreach c generate COUNT(org.apache.pig.builtin.Distinct($1.$2));
> Notice that Distinct udf is missing in Combiner and reduce stage. As a result
> distinct does not function, and incorrect results are produced.
> Distinct should have been evaluated in the 3 stages and output of Distinct
> should be given to COUNT in reduce stage.
> {code}
> # Map Reduce Plan
> #--
> MapReduce node 1-122
> Map Plan
> Local Rearrange[t
[jira] Updated: (PIG-834) incorrect plan when algebraic functions are nested
[
https://issues.apache.org/jira/browse/PIG-834?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Thejas M Nair updated PIG-834:
--
Fix Version/s: 0.7.0
> incorrect plan when algebraic functions are nested
> --
>
> Key: PIG-834
> URL: https://issues.apache.org/jira/browse/PIG-834
> Project: Pig
> Issue Type: Bug
> Components: impl
>Reporter: Thejas M Nair
>Priority: Critical
> Fix For: 0.7.0
>
>
> a = load 'students.txt' as (c1,c2,c3,c4);
> c = group a by c2;
> f = foreach c generate COUNT(org.apache.pig.builtin.Distinct($1.$2));
> Notice that Distinct udf is missing in Combiner and reduce stage. As a result
> distinct does not function, and incorrect results are produced.
> Distinct should have been evaluated in the 3 stages and output of Distinct
> should be given to COUNT in reduce stage.
> {code}
> # Map Reduce Plan
> #--
> MapReduce node 1-122
> Map Plan
> Local Rearrange[tuple]{bytearray}(false) - 1-139
> | |
> | Project[bytearray][1] - 1-140
> |
> |---New For Each(false,false)[bag] - 1-127
> | |
> | POUserFunc(org.apache.pig.builtin.COUNT$Initial)[tuple] - 1-125
> | |
> | |---POUserFunc(org.apache.pig.builtin.Distinct)[bag] - 1-126
> | |
> | |---Project[bag][2] - 1-123
> | |
> | |---Project[bag][1] - 1-124
> | |
> | Project[bytearray][0] - 1-133
> |
> |---Pre Combiner Local Rearrange[tuple]{Unknown} - 1-141
> |
>
> |---Load(hdfs://wilbur11.labs.corp.sp1.yahoo.com/user/tejas/students.txt:org.apache.pig.builtin.PigStorage)
> - 1-111
> Combine Plan
> Local Rearrange[tuple]{bytearray}(false) - 1-143
> | |
> | Project[bytearray][1] - 1-144
> |
> |---New For Each(false,false)[bag] - 1-132
> | |
> | POUserFunc(org.apache.pig.builtin.COUNT$Intermediate)[tuple] - 1-130
> | |
> | |---Project[bag][0] - 1-135
> | |
> | Project[bytearray][1] - 1-134
> |
> |---POCombinerPackage[tuple]{bytearray} - 1-137
> Reduce Plan
> Store(fakefile:org.apache.pig.builtin.PigStorage) - 1-121
> |
> |---New For Each(false)[bag] - 1-120
> | |
> | POUserFunc(org.apache.pig.builtin.COUNT$Final)[long] - 1-119
> | |
> | |---Project[bag][0] - 1-136
> |
> |---POCombinerPackage[tuple]{bytearray} - 1-145
> Global sort: false
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-834) incorrect plan when algebraic functions are nested
[
https://issues.apache.org/jira/browse/PIG-834?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Olga Natkovich updated PIG-834:
---
Priority: Major (was: Critical)
> incorrect plan when algebraic functions are nested
> --
>
> Key: PIG-834
> URL: https://issues.apache.org/jira/browse/PIG-834
> Project: Pig
> Issue Type: Bug
> Components: impl
>Reporter: Thejas M Nair
> Fix For: 0.7.0
>
>
> a = load 'students.txt' as (c1,c2,c3,c4);
> c = group a by c2;
> f = foreach c generate COUNT(org.apache.pig.builtin.Distinct($1.$2));
> Notice that Distinct udf is missing in Combiner and reduce stage. As a result
> distinct does not function, and incorrect results are produced.
> Distinct should have been evaluated in the 3 stages and output of Distinct
> should be given to COUNT in reduce stage.
> {code}
> # Map Reduce Plan
> #--
> MapReduce node 1-122
> Map Plan
> Local Rearrange[tuple]{bytearray}(false) - 1-139
> | |
> | Project[bytearray][1] - 1-140
> |
> |---New For Each(false,false)[bag] - 1-127
> | |
> | POUserFunc(org.apache.pig.builtin.COUNT$Initial)[tuple] - 1-125
> | |
> | |---POUserFunc(org.apache.pig.builtin.Distinct)[bag] - 1-126
> | |
> | |---Project[bag][2] - 1-123
> | |
> | |---Project[bag][1] - 1-124
> | |
> | Project[bytearray][0] - 1-133
> |
> |---Pre Combiner Local Rearrange[tuple]{Unknown} - 1-141
> |
>
> |---Load(hdfs://wilbur11.labs.corp.sp1.yahoo.com/user/tejas/students.txt:org.apache.pig.builtin.PigStorage)
> - 1-111
> Combine Plan
> Local Rearrange[tuple]{bytearray}(false) - 1-143
> | |
> | Project[bytearray][1] - 1-144
> |
> |---New For Each(false,false)[bag] - 1-132
> | |
> | POUserFunc(org.apache.pig.builtin.COUNT$Intermediate)[tuple] - 1-130
> | |
> | |---Project[bag][0] - 1-135
> | |
> | Project[bytearray][1] - 1-134
> |
> |---POCombinerPackage[tuple]{bytearray} - 1-137
> Reduce Plan
> Store(fakefile:org.apache.pig.builtin.PigStorage) - 1-121
> |
> |---New For Each(false)[bag] - 1-120
> | |
> | POUserFunc(org.apache.pig.builtin.COUNT$Final)[long] - 1-119
> | |
> | |---Project[bag][0] - 1-136
> |
> |---POCombinerPackage[tuple]{bytearray} - 1-145
> Global sort: false
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-834) incorrect plan when algebraic functions are nested
[
https://issues.apache.org/jira/browse/PIG-834?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashutosh Chauhan updated PIG-834:
-
Attachment: pig-834.patch
In this patch, I look for a pattern of POUserFunc followed by another
POUserFunc in the inner plan of ForEach and if thats found I flag the combiner
optimizer to not fire. This disables the combiner for this particular query
(test case included). Wondering if this fix is sufficient for this bug ?
> incorrect plan when algebraic functions are nested
> --
>
> Key: PIG-834
> URL: https://issues.apache.org/jira/browse/PIG-834
> Project: Pig
> Issue Type: Bug
> Components: impl
>Reporter: Thejas M Nair
>Assignee: Ashutosh Chauhan
> Fix For: 0.7.0
>
> Attachments: pig-834.patch
>
>
> a = load 'students.txt' as (c1,c2,c3,c4);
> c = group a by c2;
> f = foreach c generate COUNT(org.apache.pig.builtin.Distinct($1.$2));
> Notice that Distinct udf is missing in Combiner and reduce stage. As a result
> distinct does not function, and incorrect results are produced.
> Distinct should have been evaluated in the 3 stages and output of Distinct
> should be given to COUNT in reduce stage.
> {code}
> # Map Reduce Plan
> #--
> MapReduce node 1-122
> Map Plan
> Local Rearrange[tuple]{bytearray}(false) - 1-139
> | |
> | Project[bytearray][1] - 1-140
> |
> |---New For Each(false,false)[bag] - 1-127
> | |
> | POUserFunc(org.apache.pig.builtin.COUNT$Initial)[tuple] - 1-125
> | |
> | |---POUserFunc(org.apache.pig.builtin.Distinct)[bag] - 1-126
> | |
> | |---Project[bag][2] - 1-123
> | |
> | |---Project[bag][1] - 1-124
> | |
> | Project[bytearray][0] - 1-133
> |
> |---Pre Combiner Local Rearrange[tuple]{Unknown} - 1-141
> |
>
> |---Load(hdfs://wilbur11.labs.corp.sp1.yahoo.com/user/tejas/students.txt:org.apache.pig.builtin.PigStorage)
> - 1-111
> Combine Plan
> Local Rearrange[tuple]{bytearray}(false) - 1-143
> | |
> | Project[bytearray][1] - 1-144
> |
> |---New For Each(false,false)[bag] - 1-132
> | |
> | POUserFunc(org.apache.pig.builtin.COUNT$Intermediate)[tuple] - 1-130
> | |
> | |---Project[bag][0] - 1-135
> | |
> | Project[bytearray][1] - 1-134
> |
> |---POCombinerPackage[tuple]{bytearray} - 1-137
> Reduce Plan
> Store(fakefile:org.apache.pig.builtin.PigStorage) - 1-121
> |
> |---New For Each(false)[bag] - 1-120
> | |
> | POUserFunc(org.apache.pig.builtin.COUNT$Final)[long] - 1-119
> | |
> | |---Project[bag][0] - 1-136
> |
> |---POCombinerPackage[tuple]{bytearray} - 1-145
> Global sort: false
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-834) incorrect plan when algebraic functions are nested
[
https://issues.apache.org/jira/browse/PIG-834?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashutosh Chauhan updated PIG-834:
-
Attachment: pig-834_2.patch
Correct approach is following: If leaf of inner plan of ForEach is not
combinable then we dont put combiner in any case. If it is, there should not be
any other combinable POUserFunc in the ForEach's inner plan. First check
already exists in trunk. This patch checks for this second conditon and makes
sure not to fire combiner if there is any other combinable POUserFunc in the
ForEach inner plan apart from leaf POUserFunc.
> incorrect plan when algebraic functions are nested
> --
>
> Key: PIG-834
> URL: https://issues.apache.org/jira/browse/PIG-834
> Project: Pig
> Issue Type: Bug
> Components: impl
>Reporter: Thejas M Nair
>Assignee: Ashutosh Chauhan
> Fix For: 0.7.0
>
> Attachments: pig-834.patch, pig-834_2.patch
>
>
> a = load 'students.txt' as (c1,c2,c3,c4);
> c = group a by c2;
> f = foreach c generate COUNT(org.apache.pig.builtin.Distinct($1.$2));
> Notice that Distinct udf is missing in Combiner and reduce stage. As a result
> distinct does not function, and incorrect results are produced.
> Distinct should have been evaluated in the 3 stages and output of Distinct
> should be given to COUNT in reduce stage.
> {code}
> # Map Reduce Plan
> #--
> MapReduce node 1-122
> Map Plan
> Local Rearrange[tuple]{bytearray}(false) - 1-139
> | |
> | Project[bytearray][1] - 1-140
> |
> |---New For Each(false,false)[bag] - 1-127
> | |
> | POUserFunc(org.apache.pig.builtin.COUNT$Initial)[tuple] - 1-125
> | |
> | |---POUserFunc(org.apache.pig.builtin.Distinct)[bag] - 1-126
> | |
> | |---Project[bag][2] - 1-123
> | |
> | |---Project[bag][1] - 1-124
> | |
> | Project[bytearray][0] - 1-133
> |
> |---Pre Combiner Local Rearrange[tuple]{Unknown} - 1-141
> |
>
> |---Load(hdfs://wilbur11.labs.corp.sp1.yahoo.com/user/tejas/students.txt:org.apache.pig.builtin.PigStorage)
> - 1-111
> Combine Plan
> Local Rearrange[tuple]{bytearray}(false) - 1-143
> | |
> | Project[bytearray][1] - 1-144
> |
> |---New For Each(false,false)[bag] - 1-132
> | |
> | POUserFunc(org.apache.pig.builtin.COUNT$Intermediate)[tuple] - 1-130
> | |
> | |---Project[bag][0] - 1-135
> | |
> | Project[bytearray][1] - 1-134
> |
> |---POCombinerPackage[tuple]{bytearray} - 1-137
> Reduce Plan
> Store(fakefile:org.apache.pig.builtin.PigStorage) - 1-121
> |
> |---New For Each(false)[bag] - 1-120
> | |
> | POUserFunc(org.apache.pig.builtin.COUNT$Final)[long] - 1-119
> | |
> | |---Project[bag][0] - 1-136
> |
> |---POCombinerPackage[tuple]{bytearray} - 1-145
> Global sort: false
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-834) incorrect plan when algebraic functions are nested
[
https://issues.apache.org/jira/browse/PIG-834?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashutosh Chauhan updated PIG-834:
-
Status: Patch Available (was: Open)
> incorrect plan when algebraic functions are nested
> --
>
> Key: PIG-834
> URL: https://issues.apache.org/jira/browse/PIG-834
> Project: Pig
> Issue Type: Bug
> Components: impl
>Reporter: Thejas M Nair
>Assignee: Ashutosh Chauhan
> Fix For: 0.7.0
>
> Attachments: pig-834.patch, pig-834_2.patch
>
>
> a = load 'students.txt' as (c1,c2,c3,c4);
> c = group a by c2;
> f = foreach c generate COUNT(org.apache.pig.builtin.Distinct($1.$2));
> Notice that Distinct udf is missing in Combiner and reduce stage. As a result
> distinct does not function, and incorrect results are produced.
> Distinct should have been evaluated in the 3 stages and output of Distinct
> should be given to COUNT in reduce stage.
> {code}
> # Map Reduce Plan
> #--
> MapReduce node 1-122
> Map Plan
> Local Rearrange[tuple]{bytearray}(false) - 1-139
> | |
> | Project[bytearray][1] - 1-140
> |
> |---New For Each(false,false)[bag] - 1-127
> | |
> | POUserFunc(org.apache.pig.builtin.COUNT$Initial)[tuple] - 1-125
> | |
> | |---POUserFunc(org.apache.pig.builtin.Distinct)[bag] - 1-126
> | |
> | |---Project[bag][2] - 1-123
> | |
> | |---Project[bag][1] - 1-124
> | |
> | Project[bytearray][0] - 1-133
> |
> |---Pre Combiner Local Rearrange[tuple]{Unknown} - 1-141
> |
>
> |---Load(hdfs://wilbur11.labs.corp.sp1.yahoo.com/user/tejas/students.txt:org.apache.pig.builtin.PigStorage)
> - 1-111
> Combine Plan
> Local Rearrange[tuple]{bytearray}(false) - 1-143
> | |
> | Project[bytearray][1] - 1-144
> |
> |---New For Each(false,false)[bag] - 1-132
> | |
> | POUserFunc(org.apache.pig.builtin.COUNT$Intermediate)[tuple] - 1-130
> | |
> | |---Project[bag][0] - 1-135
> | |
> | Project[bytearray][1] - 1-134
> |
> |---POCombinerPackage[tuple]{bytearray} - 1-137
> Reduce Plan
> Store(fakefile:org.apache.pig.builtin.PigStorage) - 1-121
> |
> |---New For Each(false)[bag] - 1-120
> | |
> | POUserFunc(org.apache.pig.builtin.COUNT$Final)[long] - 1-119
> | |
> | |---Project[bag][0] - 1-136
> |
> |---POCombinerPackage[tuple]{bytearray} - 1-145
> Global sort: false
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-834) incorrect plan when algebraic functions are nested
[
https://issues.apache.org/jira/browse/PIG-834?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashutosh Chauhan updated PIG-834:
-
Status: Patch Available (was: Open)
Trying to get hudson going on this.
> incorrect plan when algebraic functions are nested
> --
>
> Key: PIG-834
> URL: https://issues.apache.org/jira/browse/PIG-834
> Project: Pig
> Issue Type: Bug
> Components: impl
>Reporter: Thejas M Nair
>Assignee: Ashutosh Chauhan
> Fix For: 0.7.0
>
> Attachments: pig-834.patch, pig-834_2.patch
>
>
> a = load 'students.txt' as (c1,c2,c3,c4);
> c = group a by c2;
> f = foreach c generate COUNT(org.apache.pig.builtin.Distinct($1.$2));
> Notice that Distinct udf is missing in Combiner and reduce stage. As a result
> distinct does not function, and incorrect results are produced.
> Distinct should have been evaluated in the 3 stages and output of Distinct
> should be given to COUNT in reduce stage.
> {code}
> # Map Reduce Plan
> #--
> MapReduce node 1-122
> Map Plan
> Local Rearrange[tuple]{bytearray}(false) - 1-139
> | |
> | Project[bytearray][1] - 1-140
> |
> |---New For Each(false,false)[bag] - 1-127
> | |
> | POUserFunc(org.apache.pig.builtin.COUNT$Initial)[tuple] - 1-125
> | |
> | |---POUserFunc(org.apache.pig.builtin.Distinct)[bag] - 1-126
> | |
> | |---Project[bag][2] - 1-123
> | |
> | |---Project[bag][1] - 1-124
> | |
> | Project[bytearray][0] - 1-133
> |
> |---Pre Combiner Local Rearrange[tuple]{Unknown} - 1-141
> |
>
> |---Load(hdfs://wilbur11.labs.corp.sp1.yahoo.com/user/tejas/students.txt:org.apache.pig.builtin.PigStorage)
> - 1-111
> Combine Plan
> Local Rearrange[tuple]{bytearray}(false) - 1-143
> | |
> | Project[bytearray][1] - 1-144
> |
> |---New For Each(false,false)[bag] - 1-132
> | |
> | POUserFunc(org.apache.pig.builtin.COUNT$Intermediate)[tuple] - 1-130
> | |
> | |---Project[bag][0] - 1-135
> | |
> | Project[bytearray][1] - 1-134
> |
> |---POCombinerPackage[tuple]{bytearray} - 1-137
> Reduce Plan
> Store(fakefile:org.apache.pig.builtin.PigStorage) - 1-121
> |
> |---New For Each(false)[bag] - 1-120
> | |
> | POUserFunc(org.apache.pig.builtin.COUNT$Final)[long] - 1-119
> | |
> | |---Project[bag][0] - 1-136
> |
> |---POCombinerPackage[tuple]{bytearray} - 1-145
> Global sort: false
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-834) incorrect plan when algebraic functions are nested
[
https://issues.apache.org/jira/browse/PIG-834?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashutosh Chauhan updated PIG-834:
-
Status: Open (was: Patch Available)
> incorrect plan when algebraic functions are nested
> --
>
> Key: PIG-834
> URL: https://issues.apache.org/jira/browse/PIG-834
> Project: Pig
> Issue Type: Bug
> Components: impl
>Reporter: Thejas M Nair
>Assignee: Ashutosh Chauhan
> Fix For: 0.7.0
>
> Attachments: pig-834.patch, pig-834_2.patch
>
>
> a = load 'students.txt' as (c1,c2,c3,c4);
> c = group a by c2;
> f = foreach c generate COUNT(org.apache.pig.builtin.Distinct($1.$2));
> Notice that Distinct udf is missing in Combiner and reduce stage. As a result
> distinct does not function, and incorrect results are produced.
> Distinct should have been evaluated in the 3 stages and output of Distinct
> should be given to COUNT in reduce stage.
> {code}
> # Map Reduce Plan
> #--
> MapReduce node 1-122
> Map Plan
> Local Rearrange[tuple]{bytearray}(false) - 1-139
> | |
> | Project[bytearray][1] - 1-140
> |
> |---New For Each(false,false)[bag] - 1-127
> | |
> | POUserFunc(org.apache.pig.builtin.COUNT$Initial)[tuple] - 1-125
> | |
> | |---POUserFunc(org.apache.pig.builtin.Distinct)[bag] - 1-126
> | |
> | |---Project[bag][2] - 1-123
> | |
> | |---Project[bag][1] - 1-124
> | |
> | Project[bytearray][0] - 1-133
> |
> |---Pre Combiner Local Rearrange[tuple]{Unknown} - 1-141
> |
>
> |---Load(hdfs://wilbur11.labs.corp.sp1.yahoo.com/user/tejas/students.txt:org.apache.pig.builtin.PigStorage)
> - 1-111
> Combine Plan
> Local Rearrange[tuple]{bytearray}(false) - 1-143
> | |
> | Project[bytearray][1] - 1-144
> |
> |---New For Each(false,false)[bag] - 1-132
> | |
> | POUserFunc(org.apache.pig.builtin.COUNT$Intermediate)[tuple] - 1-130
> | |
> | |---Project[bag][0] - 1-135
> | |
> | Project[bytearray][1] - 1-134
> |
> |---POCombinerPackage[tuple]{bytearray} - 1-137
> Reduce Plan
> Store(fakefile:org.apache.pig.builtin.PigStorage) - 1-121
> |
> |---New For Each(false)[bag] - 1-120
> | |
> | POUserFunc(org.apache.pig.builtin.COUNT$Final)[long] - 1-119
> | |
> | |---Project[bag][0] - 1-136
> |
> |---POCombinerPackage[tuple]{bytearray} - 1-145
> Global sort: false
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-834) incorrect plan when algebraic functions are nested
[
https://issues.apache.org/jira/browse/PIG-834?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashutosh Chauhan updated PIG-834:
-
Status: Open (was: Patch Available)
> incorrect plan when algebraic functions are nested
> --
>
> Key: PIG-834
> URL: https://issues.apache.org/jira/browse/PIG-834
> Project: Pig
> Issue Type: Bug
> Components: impl
>Reporter: Thejas M Nair
>Assignee: Ashutosh Chauhan
> Fix For: 0.7.0
>
> Attachments: pig-834.patch, pig-834_2.patch
>
>
> a = load 'students.txt' as (c1,c2,c3,c4);
> c = group a by c2;
> f = foreach c generate COUNT(org.apache.pig.builtin.Distinct($1.$2));
> Notice that Distinct udf is missing in Combiner and reduce stage. As a result
> distinct does not function, and incorrect results are produced.
> Distinct should have been evaluated in the 3 stages and output of Distinct
> should be given to COUNT in reduce stage.
> {code}
> # Map Reduce Plan
> #--
> MapReduce node 1-122
> Map Plan
> Local Rearrange[tuple]{bytearray}(false) - 1-139
> | |
> | Project[bytearray][1] - 1-140
> |
> |---New For Each(false,false)[bag] - 1-127
> | |
> | POUserFunc(org.apache.pig.builtin.COUNT$Initial)[tuple] - 1-125
> | |
> | |---POUserFunc(org.apache.pig.builtin.Distinct)[bag] - 1-126
> | |
> | |---Project[bag][2] - 1-123
> | |
> | |---Project[bag][1] - 1-124
> | |
> | Project[bytearray][0] - 1-133
> |
> |---Pre Combiner Local Rearrange[tuple]{Unknown} - 1-141
> |
>
> |---Load(hdfs://wilbur11.labs.corp.sp1.yahoo.com/user/tejas/students.txt:org.apache.pig.builtin.PigStorage)
> - 1-111
> Combine Plan
> Local Rearrange[tuple]{bytearray}(false) - 1-143
> | |
> | Project[bytearray][1] - 1-144
> |
> |---New For Each(false,false)[bag] - 1-132
> | |
> | POUserFunc(org.apache.pig.builtin.COUNT$Intermediate)[tuple] - 1-130
> | |
> | |---Project[bag][0] - 1-135
> | |
> | Project[bytearray][1] - 1-134
> |
> |---POCombinerPackage[tuple]{bytearray} - 1-137
> Reduce Plan
> Store(fakefile:org.apache.pig.builtin.PigStorage) - 1-121
> |
> |---New For Each(false)[bag] - 1-120
> | |
> | POUserFunc(org.apache.pig.builtin.COUNT$Final)[long] - 1-119
> | |
> | |---Project[bag][0] - 1-136
> |
> |---POCombinerPackage[tuple]{bytearray} - 1-145
> Global sort: false
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-834) incorrect plan when algebraic functions are nested
[
https://issues.apache.org/jira/browse/PIG-834?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashutosh Chauhan updated PIG-834:
-
Attachment: pig-834_3.patch
Instead of having recursive function walking on plan, better to have a visitor
doing that. So, this patch replaces that function with a visitor.
> incorrect plan when algebraic functions are nested
> --
>
> Key: PIG-834
> URL: https://issues.apache.org/jira/browse/PIG-834
> Project: Pig
> Issue Type: Bug
> Components: impl
>Reporter: Thejas M Nair
>Assignee: Ashutosh Chauhan
> Fix For: 0.7.0
>
> Attachments: pig-834.patch, pig-834_2.patch, pig-834_3.patch
>
>
> a = load 'students.txt' as (c1,c2,c3,c4);
> c = group a by c2;
> f = foreach c generate COUNT(org.apache.pig.builtin.Distinct($1.$2));
> Notice that Distinct udf is missing in Combiner and reduce stage. As a result
> distinct does not function, and incorrect results are produced.
> Distinct should have been evaluated in the 3 stages and output of Distinct
> should be given to COUNT in reduce stage.
> {code}
> # Map Reduce Plan
> #--
> MapReduce node 1-122
> Map Plan
> Local Rearrange[tuple]{bytearray}(false) - 1-139
> | |
> | Project[bytearray][1] - 1-140
> |
> |---New For Each(false,false)[bag] - 1-127
> | |
> | POUserFunc(org.apache.pig.builtin.COUNT$Initial)[tuple] - 1-125
> | |
> | |---POUserFunc(org.apache.pig.builtin.Distinct)[bag] - 1-126
> | |
> | |---Project[bag][2] - 1-123
> | |
> | |---Project[bag][1] - 1-124
> | |
> | Project[bytearray][0] - 1-133
> |
> |---Pre Combiner Local Rearrange[tuple]{Unknown} - 1-141
> |
>
> |---Load(hdfs://wilbur11.labs.corp.sp1.yahoo.com/user/tejas/students.txt:org.apache.pig.builtin.PigStorage)
> - 1-111
> Combine Plan
> Local Rearrange[tuple]{bytearray}(false) - 1-143
> | |
> | Project[bytearray][1] - 1-144
> |
> |---New For Each(false,false)[bag] - 1-132
> | |
> | POUserFunc(org.apache.pig.builtin.COUNT$Intermediate)[tuple] - 1-130
> | |
> | |---Project[bag][0] - 1-135
> | |
> | Project[bytearray][1] - 1-134
> |
> |---POCombinerPackage[tuple]{bytearray} - 1-137
> Reduce Plan
> Store(fakefile:org.apache.pig.builtin.PigStorage) - 1-121
> |
> |---New For Each(false)[bag] - 1-120
> | |
> | POUserFunc(org.apache.pig.builtin.COUNT$Final)[long] - 1-119
> | |
> | |---Project[bag][0] - 1-136
> |
> |---POCombinerPackage[tuple]{bytearray} - 1-145
> Global sort: false
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-834) incorrect plan when algebraic functions are nested
[
https://issues.apache.org/jira/browse/PIG-834?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashutosh Chauhan updated PIG-834:
-
Status: Patch Available (was: Open)
> incorrect plan when algebraic functions are nested
> --
>
> Key: PIG-834
> URL: https://issues.apache.org/jira/browse/PIG-834
> Project: Pig
> Issue Type: Bug
> Components: impl
>Reporter: Thejas M Nair
>Assignee: Ashutosh Chauhan
> Fix For: 0.7.0
>
> Attachments: pig-834.patch, pig-834_2.patch, pig-834_3.patch
>
>
> a = load 'students.txt' as (c1,c2,c3,c4);
> c = group a by c2;
> f = foreach c generate COUNT(org.apache.pig.builtin.Distinct($1.$2));
> Notice that Distinct udf is missing in Combiner and reduce stage. As a result
> distinct does not function, and incorrect results are produced.
> Distinct should have been evaluated in the 3 stages and output of Distinct
> should be given to COUNT in reduce stage.
> {code}
> # Map Reduce Plan
> #--
> MapReduce node 1-122
> Map Plan
> Local Rearrange[tuple]{bytearray}(false) - 1-139
> | |
> | Project[bytearray][1] - 1-140
> |
> |---New For Each(false,false)[bag] - 1-127
> | |
> | POUserFunc(org.apache.pig.builtin.COUNT$Initial)[tuple] - 1-125
> | |
> | |---POUserFunc(org.apache.pig.builtin.Distinct)[bag] - 1-126
> | |
> | |---Project[bag][2] - 1-123
> | |
> | |---Project[bag][1] - 1-124
> | |
> | Project[bytearray][0] - 1-133
> |
> |---Pre Combiner Local Rearrange[tuple]{Unknown} - 1-141
> |
>
> |---Load(hdfs://wilbur11.labs.corp.sp1.yahoo.com/user/tejas/students.txt:org.apache.pig.builtin.PigStorage)
> - 1-111
> Combine Plan
> Local Rearrange[tuple]{bytearray}(false) - 1-143
> | |
> | Project[bytearray][1] - 1-144
> |
> |---New For Each(false,false)[bag] - 1-132
> | |
> | POUserFunc(org.apache.pig.builtin.COUNT$Intermediate)[tuple] - 1-130
> | |
> | |---Project[bag][0] - 1-135
> | |
> | Project[bytearray][1] - 1-134
> |
> |---POCombinerPackage[tuple]{bytearray} - 1-137
> Reduce Plan
> Store(fakefile:org.apache.pig.builtin.PigStorage) - 1-121
> |
> |---New For Each(false)[bag] - 1-120
> | |
> | POUserFunc(org.apache.pig.builtin.COUNT$Final)[long] - 1-119
> | |
> | |---Project[bag][0] - 1-136
> |
> |---POCombinerPackage[tuple]{bytearray} - 1-145
> Global sort: false
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-834) incorrect plan when algebraic functions are nested
[
https://issues.apache.org/jira/browse/PIG-834?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashutosh Chauhan updated PIG-834:
-
Resolution: Fixed
Status: Resolved (was: Patch Available)
Patch checked-in.
> incorrect plan when algebraic functions are nested
> --
>
> Key: PIG-834
> URL: https://issues.apache.org/jira/browse/PIG-834
> Project: Pig
> Issue Type: Bug
> Components: impl
>Reporter: Thejas M Nair
>Assignee: Ashutosh Chauhan
> Fix For: 0.7.0
>
> Attachments: pig-834.patch, pig-834_2.patch, pig-834_3.patch
>
>
> a = load 'students.txt' as (c1,c2,c3,c4);
> c = group a by c2;
> f = foreach c generate COUNT(org.apache.pig.builtin.Distinct($1.$2));
> Notice that Distinct udf is missing in Combiner and reduce stage. As a result
> distinct does not function, and incorrect results are produced.
> Distinct should have been evaluated in the 3 stages and output of Distinct
> should be given to COUNT in reduce stage.
> {code}
> # Map Reduce Plan
> #--
> MapReduce node 1-122
> Map Plan
> Local Rearrange[tuple]{bytearray}(false) - 1-139
> | |
> | Project[bytearray][1] - 1-140
> |
> |---New For Each(false,false)[bag] - 1-127
> | |
> | POUserFunc(org.apache.pig.builtin.COUNT$Initial)[tuple] - 1-125
> | |
> | |---POUserFunc(org.apache.pig.builtin.Distinct)[bag] - 1-126
> | |
> | |---Project[bag][2] - 1-123
> | |
> | |---Project[bag][1] - 1-124
> | |
> | Project[bytearray][0] - 1-133
> |
> |---Pre Combiner Local Rearrange[tuple]{Unknown} - 1-141
> |
>
> |---Load(hdfs://wilbur11.labs.corp.sp1.yahoo.com/user/tejas/students.txt:org.apache.pig.builtin.PigStorage)
> - 1-111
> Combine Plan
> Local Rearrange[tuple]{bytearray}(false) - 1-143
> | |
> | Project[bytearray][1] - 1-144
> |
> |---New For Each(false,false)[bag] - 1-132
> | |
> | POUserFunc(org.apache.pig.builtin.COUNT$Intermediate)[tuple] - 1-130
> | |
> | |---Project[bag][0] - 1-135
> | |
> | Project[bytearray][1] - 1-134
> |
> |---POCombinerPackage[tuple]{bytearray} - 1-137
> Reduce Plan
> Store(fakefile:org.apache.pig.builtin.PigStorage) - 1-121
> |
> |---New For Each(false)[bag] - 1-120
> | |
> | POUserFunc(org.apache.pig.builtin.COUNT$Final)[long] - 1-119
> | |
> | |---Project[bag][0] - 1-136
> |
> |---POCombinerPackage[tuple]{bytearray} - 1-145
> Global sort: false
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
