[
https://issues.apache.org/jira/browse/PIG-3287?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13770022#comment-13770022
]
Christon DeWan commented on PIG-3287:
-------------------------------------
This is a very long workflow and needs both optimizations to work effectively.
For now I've refactored my flow to avoid needing both at once.
> MultiQueryOptimizer can prevent CombinerOptimizer from working
> --------------------------------------------------------------
>
> Key: PIG-3287
> URL: https://issues.apache.org/jira/browse/PIG-3287
> Project: Pig
> Issue Type: Bug
> Affects Versions: 0.10.1
> Reporter: Christon DeWan
>
> The CombinerOptimizer does not operate on the script below. As a result, all
> work is done in the reducer(s), killing performance. Removing one STORE or
> refactoring the query to use a single FOREACH after the group allows the
> CombinerOptimizer to work.
> {noformat}
> %declare DUMMY `bash -c '(for (( i=0; \$i < 10; i++ )); do echo \$i 5; done)
> | hadoop fs -put - /tmp/test_data.tsv; true'`
> s = LOAD '/tmp/test_data.tsv' USING PigStorage(' ') AS (n:long, g:long);
> grouped = GROUP s BY g;
> counted = FOREACH grouped GENERATE flatten($0), COUNT_STAR($1);
> STORE counted INTO '/tmp/test_count';
> summed = FOREACH grouped GENERATE flatten($0), SUM($1.n);
> STORE summed INTO '/tmp/test_sum';
> FS -rmr /tmp/test_{data.tsv,count,sum}
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
