[
https://issues.apache.org/jira/browse/HIVE-2621?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13174671#comment-13174671
]
Phabricator commented on HIVE-2621:
-----------------------------------
njain has commented on the revision "HIVE-2621 [jira] Allow multiple group bys
with the same input data and spray keys to be run on the same reducer.".
INLINE COMMENTS
ql/src/test/queries/clientpositive/groupby7_noskew_multi_single_reducer.q:12
This does not look right.
We would like to make hive.multigroupby.singlereducer as true by default.
But, we are un-necessarily generating 3 MR jobs for this query (with no
distinct). I think, we can get it in 2 MR jobs today (not 100% sure)
ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java:6273 It
would be good to merge the code path with the above if block
(optimizeMultiGroupBy).
The common distinct expression should return the common distinct
checking for the parameter HIVEMULTIGROUPBYSINGLEREDUCER.
Or, it might be simpler to remove the above if block (the
optimizeMultiGroupby should be covered by this block).
Anyway, the above if block (6253-6272) seems broken
ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java:6211 I
think this code can be simplified.
The function getCommonDistinctExprs can be removed
REVISION DETAIL
https://reviews.facebook.net/D567
> Allow multiple group bys with the same input data and spray keys to be run on
> the same reducer.
> -----------------------------------------------------------------------------------------------
>
> Key: HIVE-2621
> URL: https://issues.apache.org/jira/browse/HIVE-2621
> Project: Hive
> Issue Type: New Feature
> Reporter: Kevin Wilfong
> Assignee: Kevin Wilfong
> Attachments: HIVE-2621.1.patch.txt, HIVE-2621.D567.1.patch,
> HIVE-2621.D567.2.patch, HIVE-2621.D567.3.patch
>
>
> Currently, when a user runs a query, such as a multi-insert, where each
> insertion subclause consists of a simple query followed by a group by, the
> group bys for each clause are run on a separate reducer. This requires
> writing the data for each group by clause to an intermediate file, and then
> reading it back. This uses a significant amount of the total CPU consumed by
> the query for an otherwise simple query.
> If the subclauses are grouped by their distinct expressions and group by
> keys, with all of the group by expressions for a group of subclauses run on a
> single reducer, this would reduce the amount of reading/writing to
> intermediate files for some queries.
> To do this, for each group of subclauses, in the mapper we would execute a
> the filters for each subclause 'or'd together (provided each subclause has a
> filter) followed by a reduce sink. In the reducer, the child operators would
> be each subclauses filter followed by the group by and any subsequent
> operations.
> Note that this would require turning off map aggregation, so we would need to
> make using this type of plan configurable.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
