[ https://issues.apache.org/jira/browse/HIVE-2621?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Phabricator updated HIVE-2621: ------------------------------ Attachment: HIVE-2621.D567.1.patch kevinwilfong requested code review of "HIVE-2621 [jira] Allow multiple group bys with the same input data and spray keys to be run on the same reducer.". Reviewers: JIRA The meaningful changes are all in how the plan is generated. If the conf variable has been set, the subclauses are first grouped by their group by keys and distinct keys. To facilitate this I added a wrapper class to ExprNodeDesc which makes equals like the isSame method. If the conf variable is not set, I create a single group of all the subqueries. Then, provided certain conditions are met, e.g. the conf variable is set, there is a group by and there are aggregations, the skew conf variable hasn't been set, I create the new plan for each group, otherwise the old plan is produced. To start I generate the common filter by 'or'ing the group's clauses' filters. This goes into a select operator, which goes into a new reduce operator. The reduce operator is like the typical 1 MR group by reduce operator, except that to generate the reduce values it loops over each of the group's subclauses' aggregations and the columns used in the where clauses. This goes into a forward operator and for each subclause the forward operator has a child filter operator, if the subclause has a filter, and a group by operator. Each group by operator is followed by the operators which would normally follow it in a plan. TEST PLAN I added some unit tests. I verified these unit tests and the old unit tests all passed. I created a sample query which consisted of a multi-insert from a table with 1,000,000 rows, going into 6 tables, each of which's subclause consisted of a group by, and a count distinct, as well as some other aggregations and havings. The subclauses were constructed such that they could be grouped into two reducers using the new plan. I also ensured that the data was such that map aggregation was turned of early using the existing plan. I verified that this query saw a significant improvement in its CPU usage. REVISION DETAIL https://reviews.facebook.net/D567 AFFECTED FILES conf/hive-default.xml common/src/java/org/apache/hadoop/hive/conf/HiveConf.java ql/src/test/results/clientpositive/groupby7_noskew_multi_single_reducer.q.out ql/src/test/results/clientpositive/groupby_multi_single_reducer.q.out ql/src/test/results/clientpositive/groupby_complex_types_multi_single_reducer.q.out ql/src/test/queries/clientpositive/groupby_multi_single_reducer.q ql/src/test/queries/clientpositive/groupby7_noskew_multi_single_reducer.q ql/src/test/queries/clientpositive/groupby_complex_types_multi_single_reducer.q ql/src/java/org/apache/hadoop/hive/ql/plan/ExprNodeDesc.java ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java MANAGE HERALD DIFFERENTIAL RULES https://reviews.facebook.net/herald/view/differential/ WHY DID I GET THIS EMAIL? https://reviews.facebook.net/herald/transcript/1269/ Tip: use the X-Herald-Rules header to filter Herald messages in your client. > Allow multiple group bys with the same input data and spray keys to be run on > the same reducer. > ----------------------------------------------------------------------------------------------- > > Key: HIVE-2621 > URL: https://issues.apache.org/jira/browse/HIVE-2621 > Project: Hive > Issue Type: New Feature > Reporter: Kevin Wilfong > Assignee: Kevin Wilfong > Attachments: HIVE-2621.1.patch.txt, HIVE-2621.D567.1.patch > > > Currently, when a user runs a query, such as a multi-insert, where each > insertion subclause consists of a simple query followed by a group by, the > group bys for each clause are run on a separate reducer. This requires > writing the data for each group by clause to an intermediate file, and then > reading it back. This uses a significant amount of the total CPU consumed by > the query for an otherwise simple query. > If the subclauses are grouped by their distinct expressions and group by > keys, with all of the group by expressions for a group of subclauses run on a > single reducer, this would reduce the amount of reading/writing to > intermediate files for some queries. > To do this, for each group of subclauses, in the mapper we would execute a > the filters for each subclause 'or'd together (provided each subclause has a > filter) followed by a reduce sink. In the reducer, the child operators would > be each subclauses filter followed by the group by and any subsequent > operations. > Note that this would require turning off map aggregation, so we would need to > make using this type of plan configurable. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira