[jira] [Commented] (PIG-4767) Partition filter not pushed down when filter clause references variable from another load path

2017-07-19 Thread Koji Noguchi (JIRA) 

[ 
https://issues.apache.org/jira/browse/PIG-4767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16093482#comment-16093482
 ] 

Koji Noguchi commented on PIG-4767:
---

Can we push this to 0.17 branch?

> Partition filter not pushed down when filter clause references variable from 
> another load path
> --
>
> Key: PIG-4767
> URL: https://issues.apache.org/jira/browse/PIG-4767
> Project: Pig
>  Issue Type: Bug
>Affects Versions: 0.15.0
>Reporter: Anthony Hsu
>Assignee: Koji Noguchi
> Fix For: 0.18.0
>
> Attachments: pig-4767-v01.patch
>
>
> To reproduce:
> {noformat:title=test.pig}
> a = load 'a.txt';
> a_group = group a all;
> a_count = foreach a_group generate COUNT(a) as count;
> b = load 'mytable' using org.apache.hcatalog.pig.HCatLoader();
> b = filter b by datepartition == '2015-09-01-00' and foo == a_count.count;
> dump b;
> {noformat}
> The above query ends up reading all the table partitions. If you remove the 
> {{foo == a_count.count}} clause or replace {{a_count.count}} with a constant, 
> then partition filtering happens properly.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (PIG-4767) Partition filter not pushed down when filter clause references variable from another load path

2017-07-14 Thread Anthony Hsu (JIRA) 

[ 
https://issues.apache.org/jira/browse/PIG-4767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16087938#comment-16087938
 ] 

Anthony Hsu commented on PIG-4767:
--

No problem, [~knoguchi]. Thanks for the fix!

> Partition filter not pushed down when filter clause references variable from 
> another load path
> --
>
> Key: PIG-4767
> URL: https://issues.apache.org/jira/browse/PIG-4767
> Project: Pig
>  Issue Type: Bug
>Affects Versions: 0.15.0
>Reporter: Anthony Hsu
>Assignee: Koji Noguchi
> Fix For: 0.18.0
>
> Attachments: pig-4767-v01.patch
>
>
> To reproduce:
> {noformat:title=test.pig}
> a = load 'a.txt';
> a_group = group a all;
> a_count = foreach a_group generate COUNT(a) as count;
> b = load 'mytable' using org.apache.hcatalog.pig.HCatLoader();
> b = filter b by datepartition == '2015-09-01-00' and foo == a_count.count;
> dump b;
> {noformat}
> The above query ends up reading all the table partitions. If you remove the 
> {{foo == a_count.count}} clause or replace {{a_count.count}} with a constant, 
> then partition filtering happens properly.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (PIG-4767) Partition filter not pushed down when filter clause references variable from another load path

2017-07-13 Thread Koji Noguchi (JIRA) 

[ 
https://issues.apache.org/jira/browse/PIG-4767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16086170#comment-16086170
 ] 

Koji Noguchi commented on PIG-4767:
---

Forgot to mention, Anthony, sorry for taking this long to fix this bug.  

> Partition filter not pushed down when filter clause references variable from 
> another load path
> --
>
> Key: PIG-4767
> URL: https://issues.apache.org/jira/browse/PIG-4767
> Project: Pig
>  Issue Type: Bug
>Affects Versions: 0.15.0
>Reporter: Anthony Hsu
>Assignee: Koji Noguchi
> Fix For: 0.18.0
>
> Attachments: pig-4767-v01.patch
>
>
> To reproduce:
> {noformat:title=test.pig}
> a = load 'a.txt';
> a_group = group a all;
> a_count = foreach a_group generate COUNT(a) as count;
> b = load 'mytable' using org.apache.hcatalog.pig.HCatLoader();
> b = filter b by datepartition == '2015-09-01-00' and foo == a_count.count;
> dump b;
> {noformat}
> The above query ends up reading all the table partitions. If you remove the 
> {{foo == a_count.count}} clause or replace {{a_count.count}} with a constant, 
> then partition filtering happens properly.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (PIG-4767) Partition filter not pushed down when filter clause references variable from another load path

2017-07-12 Thread Daniel Dai (JIRA) 

[ 
https://issues.apache.org/jira/browse/PIG-4767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16084758#comment-16084758
 ] 

Daniel Dai commented on PIG-4767:
-

That's right, PartitionFilterOptimizer and PredicatePushdownOptimizer does not 
push filter up. The problem PIG-1669 try to solve does not exist. +1.

> Partition filter not pushed down when filter clause references variable from 
> another load path
> --
>
> Key: PIG-4767
> URL: https://issues.apache.org/jira/browse/PIG-4767
> Project: Pig
>  Issue Type: Bug
>Affects Versions: 0.15.0
>Reporter: Anthony Hsu
>Assignee: Koji Noguchi
> Fix For: 0.18.0
>
> Attachments: pig-4767-v01.patch
>
>
> To reproduce:
> {noformat:title=test.pig}
> a = load 'a.txt';
> a_group = group a all;
> a_count = foreach a_group generate COUNT(a) as count;
> b = load 'mytable' using org.apache.hcatalog.pig.HCatLoader();
> b = filter b by datepartition == '2015-09-01-00' and foo == a_count.count;
> dump b;
> {noformat}
> The above query ends up reading all the table partitions. If you remove the 
> {{foo == a_count.count}} clause or replace {{a_count.count}} with a constant, 
> then partition filtering happens properly.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (PIG-4767) Partition filter not pushed down when filter clause references variable from another load path

2017-07-11 Thread Anthony Hsu (JIRA) 

[ 
https://issues.apache.org/jira/browse/PIG-4767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16082502#comment-16082502
 ] 

Anthony Hsu commented on PIG-4767:
--

Thanks for the patch, [~knoguchi]! I tested it out and it solves my problem.

> Partition filter not pushed down when filter clause references variable from 
> another load path
> --
>
> Key: PIG-4767
> URL: https://issues.apache.org/jira/browse/PIG-4767
> Project: Pig
>  Issue Type: Bug
>Affects Versions: 0.15.0
>Reporter: Anthony Hsu
>Assignee: Koji Noguchi
> Fix For: 0.18.0
>
> Attachments: pig-4767-v01.patch
>
>
> To reproduce:
> {noformat:title=test.pig}
> a = load 'a.txt';
> a_group = group a all;
> a_count = foreach a_group generate COUNT(a) as count;
> b = load 'mytable' using org.apache.hcatalog.pig.HCatLoader();
> b = filter b by datepartition == '2015-09-01-00' and foo == a_count.count;
> dump b;
> {noformat}
> The above query ends up reading all the table partitions. If you remove the 
> {{foo == a_count.count}} clause or replace {{a_count.count}} with a constant, 
> then partition filtering happens properly.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)