-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65174/
-----------------------------------------------------------
(Updated 7 11, 2018, 12:30 오후)
Review request for hive.
Changes
-------
Removed plan optimizer part and updated the output files.
Bugs: HIVE-17896
https://issues.apache.org/jira/browse/HIVE-17896
Repository: hive-git
Description
-------
For TPC-DS Query27, the TopN operation is delayed by the group-by - the
group-by operator buffers up all the rows before discarding the 99% of the rows
in the TopN Hash within the ReduceSink Operator.
The RS TopN operator is very restrictive as it only supports doing the
filtering on the shuffle keys, but it is better to do this before breaking the
vectors into rows and losing the isRepeating properties.
Adding a TopN Key operator in the physical operator tree allows the following
to happen.
GBY->RS(Top=1)
can become
TNK(1)->GBY->RS(Top=1)
So that, the TopNKey can remove rows before they are buffered into the GBY and
consume memory.
Here's the equivalent implementation in Presto
https://github.com/prestodb/presto/blob/master/presto-main/src/main/java/com/facebook/presto/operator/TopNOperator.java#L35
Adding this as a sub-feature of GroupBy prevents further optimizations if the
GBY is on keys "a,b,c" and the TopNKey is on just "a".
Diffs (updated)
-----
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 6ea68c3500
itests/src/test/resources/testconfiguration.properties 9e012ce2f8
ql/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/ql/plan/api/OperatorType.java
a002348013
ql/src/java/org/apache/hadoop/hive/ql/exec/KeyWrapperFactory.java 71ee25d9e0
ql/src/java/org/apache/hadoop/hive/ql/exec/OperatorFactory.java 7bb6590d5e
ql/src/java/org/apache/hadoop/hive/ql/exec/TopNKeyOperator.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorTopNKeyOperator.java
PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/optimizer/TopNKeyProcessor.java
PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/Vectorizer.java
7afbf04797
ql/src/java/org/apache/hadoop/hive/ql/parse/TezCompiler.java dfd790853b
ql/src/java/org/apache/hadoop/hive/ql/plan/TopNKeyDesc.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/plan/VectorTopNKeyDesc.java
PRE-CREATION
ql/src/test/queries/clientpositive/topnkey.q PRE-CREATION
ql/src/test/queries/clientpositive/vector_topnkey.q PRE-CREATION
ql/src/test/results/clientpositive/llap/topnkey.q.out PRE-CREATION
ql/src/test/results/clientpositive/llap/vector_topnkey.q.out PRE-CREATION
ql/src/test/results/clientpositive/tez/topnkey.q.out PRE-CREATION
ql/src/test/results/clientpositive/tez/vector_topnkey.q.out PRE-CREATION
ql/src/test/results/clientpositive/topnkey.q.out PRE-CREATION
ql/src/test/results/clientpositive/vector_topnkey.q.out PRE-CREATION
serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/ObjectInspectorUtils.java
9393fb853f
Diff: https://reviews.apache.org/r/65174/diff/3/
Changes: https://reviews.apache.org/r/65174/diff/2-3/
Testing
-------
Thanks,
Teddy Choi
