-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/64912/
-----------------------------------------------------------
Review request for hive.
Bugs: HIVE-17896
https://issues.apache.org/jira/browse/HIVE-17896
Repository: hive-git
Description
-------
For TPC-DS Query27, the TopN operation is delayed by the group-by - the
group-by operator buffers up all the rows before discarding the 99% of the rows
in the TopN Hash within the ReduceSink Operator.
The RS TopN operator is very restrictive as it only supports doing the
filtering on the shuffle keys, but it is better to do this before breaking the
vectors into rows and losing the isRepeating properties.
Adding a TopN Key operator in the physical operator tree allows the following
to happen.
GBY->RS(Top=1)
can become
TNK(1)->GBY->RS(Top=1)
So that, the TopNKey can remove rows before they are buffered into the GBY and
consume memory.
Adding this as a sub-feature of GroupBy prevents further optimizations if the
GBY is on keys "a,b,c" and the TopNKey is on just "a".
Diffs
-----
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java be83489cb3
itests/src/test/resources/testconfiguration.properties 9121ca8d1e
ql/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/ql/plan/api/OperatorType.java
a002348013
ql/src/java/org/apache/hadoop/hive/ql/exec/KeyWrapperFactory.java 775c737bba
ql/src/java/org/apache/hadoop/hive/ql/exec/OperatorFactory.java e665064133
ql/src/java/org/apache/hadoop/hive/ql/exec/TopNKeyOperator.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorTopNKeyOperator.java
PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/optimizer/TopNKeyProcessor.java
PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/optimizer/TopNKeyPushdown.java
PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/Vectorizer.java
6f289703a7
ql/src/java/org/apache/hadoop/hive/ql/parse/TezCompiler.java be33f38003
ql/src/java/org/apache/hadoop/hive/ql/plan/TopNKeyDesc.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/plan/VectorTopNKeyDesc.java
PRE-CREATION
ql/src/test/queries/clientpositive/topnkey.q PRE-CREATION
ql/src/test/queries/clientpositive/vector_topnkey.q PRE-CREATION
ql/src/test/results/clientpositive/llap/topnkey.q.out PRE-CREATION
ql/src/test/results/clientpositive/llap/vector_topnkey.q.out PRE-CREATION
ql/src/test/results/clientpositive/tez/topnkey.q.out PRE-CREATION
ql/src/test/results/clientpositive/tez/vector_topnkey.q.out PRE-CREATION
ql/src/test/results/clientpositive/topnkey.q.out PRE-CREATION
ql/src/test/results/clientpositive/vector_topnkey.q.out PRE-CREATION
serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/ObjectInspectorUtils.java
698ebe1ffe
Diff: https://reviews.apache.org/r/64912/diff/1/
Testing
-------
Thanks,
Teddy Choi
