[
https://issues.apache.org/jira/browse/PIG-3395?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Cheolsoo Park updated PIG-3395:
-------------------------------
Attachment: thread_dump.txt
Attaching the tread dump. I will shortly upload a patch that refactors
PColFilterExtractor that makes it work.
> Large filter expression makes Pig hang
> --------------------------------------
>
> Key: PIG-3395
> URL: https://issues.apache.org/jira/browse/PIG-3395
> Project: Pig
> Issue Type: Bug
> Components: impl
> Reporter: Cheolsoo Park
> Assignee: Cheolsoo Park
> Fix For: 0.12
>
> Attachments: thread_dump.txt
>
>
> Currently, partition filter push down is quite costly. For example, if you
> have many nested or/and expressions, Pig hangs:
> {code}
> base = load '<partitioned table>' using MyStorage();
> filt = filter base by
> (dateint == 20130719 and batchid == 'merged_1' and hour IN (19,20,21,22,23))
> or
> (dateint == 20130720 and batchid == 'merged_1' and hour IN
> (0,1,2,3,4,5,6,7,8))
> or
> (dateint == 20130720 and batchid == 'merged_2' and hour == 7)
> or
> (dateint == 20130720 and batchid == 'merged_1' and hour IN
> (9,10,11,12,13,14,15,16,17,18,19,20,21,22,23))
> or
> (dateint == 20130721 and batchid == 'merged_1' and hour IN
> (0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23))
> or
> (dateint == 20130722 and batchid == 'merged_1' and hour IN
> (0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16));
> dump filt;
> {code}
> Note that IN operator is converted to nested OR's by Pig parser.
> Looking at the thread dump, I found it creates almost 60 stack frames and
> makes JVM suffer. (I will attach full stack trace.)
> {code}
> <repeated ...>
> at
> org.apache.pig.newplan.PColFilterExtractor.visit(PColFilterExtractor.java:504)
> at
> org.apache.pig.newplan.PColFilterExtractor.visit(PColFilterExtractor.java:237)
> at
> org.apache.pig.newplan.PColFilterExtractor.visit(PColFilterExtractor.java:504)
> at
> org.apache.pig.newplan.PColFilterExtractor.visit(PColFilterExtractor.java:214)
> at
> org.apache.pig.newplan.PColFilterExtractor.visit(PColFilterExtractor.java:504)
> at
> org.apache.pig.newplan.PColFilterExtractor.visit(PColFilterExtractor.java:211)
> at
> org.apache.pig.newplan.PColFilterExtractor.visit(PColFilterExtractor.java:108)
> {code}
> Although the filter expression can be simplified, it seems possible to make
> PColFilterExtractor more efficient.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
