[ https://issues.apache.org/jira/browse/PIG-1637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12915880#action_12915880 ]
Daniel Dai commented on PIG-1637: --------------------------------- test-patch result for PIG-1637-2.patch: [exec] +1 overall. [exec] [exec] +1 @author. The patch does not contain any @author tags. [exec] [exec] +1 tests included. The patch appears to include 3 new or modified tests. [exec] [exec] +1 javadoc. The javadoc tool did not generate any warning messages. [exec] [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings. [exec] [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings. [exec] [exec] +1 release audit. The applied patch does not increase the total number of release audit warnings. > Combiner not use because optimizor inserts a foreach between group and > algebric function > ---------------------------------------------------------------------------------------- > > Key: PIG-1637 > URL: https://issues.apache.org/jira/browse/PIG-1637 > Project: Pig > Issue Type: Bug > Affects Versions: 0.8.0 > Reporter: Daniel Dai > Assignee: Daniel Dai > Fix For: 0.8.0 > > Attachments: PIG-1637-1.patch, PIG-1637-2.patch > > > The following script does not use combiner after new optimization change. > {code} > A = load ':INPATH:/pigmix/page_views' using > org.apache.pig.test.udf.storefunc.PigPerformanceLoader() > as (user, action, timespent, query_term, ip_addr, timestamp, > estimated_revenue, page_info, page_links); > B = foreach A generate user, (int)timespent as timespent, > (double)estimated_revenue as estimated_revenue; > C = group B all; > D = foreach C generate SUM(B.timespent), AVG(B.estimated_revenue); > store D into ':OUTPATH:'; > {code} > This is because after group, optimizer detect group key is not used > afterward, it add a foreach statement after C. This is how it looks like > after optimization: > {code} > A = load ':INPATH:/pigmix/page_views' using > org.apache.pig.test.udf.storefunc.PigPerformanceLoader() > as (user, action, timespent, query_term, ip_addr, timestamp, > estimated_revenue, page_info, page_links); > B = foreach A generate user, (int)timespent as timespent, > (double)estimated_revenue as estimated_revenue; > C = group B all; > C1 = foreach C generate B; > D = foreach C1 generate SUM(B.timespent), AVG(B.estimated_revenue); > store D into ':OUTPATH:'; > {code} > That cancel the combiner optimization for D. > The way to solve the issue is to merge the C1 we inserted and D. Currently, > we do not merge these two foreach. The reason is that one output of the first > foreach (B) is referred twice in D, and currently rule assume after merge, we > need to calculate B twice in D. Actually, C1 is only doing projection, no > calculation of B. Merging C1 and D will not result calculating B twice. So C1 > and D should be merged. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.