[
https://issues.apache.org/jira/browse/PIG-984?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12764290#action_12764290
]
Hadoop QA commented on PIG-984:
-------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12421781/PIG-984.patch
against trunk revision 823693.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 3 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
-1 javac. The applied patch generated 413 javac compiler warnings (more
than the trunk's current 411 warnings).
-1 findbugs. The patch appears to introduce 3 new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of
release audit warnings.
+1 core tests. The patch passed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
Test results:
http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/69/testReport/
Findbugs warnings:
http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/69/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output:
http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/69/console
This message is automatically generated.
> PERFORMANCE: Implement a map-side group operator to speed up processing of
> ordered data
> ----------------------------------------------------------------------------------------
>
> Key: PIG-984
> URL: https://issues.apache.org/jira/browse/PIG-984
> Project: Pig
> Issue Type: New Feature
> Reporter: Richard Ding
> Assignee: Richard Ding
> Attachments: PIG-984.patch
>
>
> The general group by operation in Pig needs both mappers and reducers (the
> aggregation is done in reducers). This incurs disk writes/reads between
> mappers and reducers.
> However, in the cases where the input data has the following properties
> 1. The records with the same key are grouped together (such as the data is
> sorted by the keys).
> 2. The records with the same key are in the same mapper input.
> the group by operation can be performed in the mappers only and thus remove
> the overhead of disk writes/reads.
> Alan proposed adding a hint to the group by clause like this one:
> {code}
> A = load 'input' using SomeLoader(...);
> B = group A by $0 using "mapside";
> C = foreach B generate ...
> {code}
> The proposed addition of using "mapside" to group will be a mapside group
> operator that collects all records for a given key into a buffer. When it
> sees a key change it will emit the key and bag for records it had buffered.
> It will assume that all keys for a given record are collected together and
> thus there is not need to buffer across keys.
> It is expected that "SomeLoader" will be implemented by data systems such as
> Zebra to ensure the data emitted by the loader satisfies the above properties
> (1) and (2).
> It will be the responsibility of the user (or the loader) to guarantee these
> properties (1) & (2) before invoking the mapside hint for the group by
> clause. The Pig runtime can't check for the errors in the input data.
> For the group by clauses with mapside hint, Pig Latin will only support group
> by columns (including *), not group by expressions nor group all.
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
